A new survey has identified resourcing issues as the top barrier preventing businesses from successfully managing and exploiting the possibilities opened up to them by Big Data solutions.

The research, carried out in Spring 2014 and sponsored by Talend, found that nearly two out of every five (39%) organisations cited a lack of skills or a lack of time as the main type of difficulty standing in the way of them successfully managing and exploiting the potential opportunities afforded by Big Data.

“The skills issue is one of the biggest “brakes” on business benefiting from Big Data today,” said Yves de Montcheuil, VP Marketing, Talend. “As the Big Data environment has evolved, the technology required to navigate it has become increasingly complex and the pool of engineers capable of using it or even understanding it has not followed suit.

“This means there are few staff with Big Data skills and expertise on the job market, and the few that are available, are typically extortionately expensive to hire,” adds de Montcheuil. “So, businesses need to develop expertise internally and train staff in Big Data skills but even then they face the risk that once trained those staff will defect to a competitor prepared to pay them more money.”

The survey did find that in many cases the IT department takes the lead on Big Data. 28% said the initial demand to investigate Big Data solutions came from IT, more than twice the proportion (13%) who cited the board of directors/senior management.

Yet, this focus on IT exacerbates the resourcing issue rather than solving it. In most businesses, IT departments have neither the resource nor the power and influence to look at Big Data strategically. Instead, Big Data is often seen merely as an add-on to the daily routine of ‘keeping the lights on’. 37% of the sample said “our IT department is looking at Big Data opportunities as well as performing day to day operational IT functions”. In contrast, just 24% said the two activities are separated. 

Many businesses are currently unhappy with the status quo in terms of the resource targeted at Big Data. Indeed, over half the sample (55%) support the view that to exploit Big Data to its full potential, there is a need to create multi-disciplinary teams (including IT, business heads, analysts, industry experts etc.)

The research also identified the culture of businesses, and more especially the vision emanating from the top of the company down, as key to whether organisations are positioned to take advantage of Big Data opportunities. Advanced respondents (those who had begun to use Big Data analytics) had a much more positive vision overall of what data could bring them.

44% of advanced users, for example, said: “Our overall strategy, driven by the business, is to find new opportunities by making the most of our data” compared with just 28% of less advanced firms. 

Companies further back in the Big Data journey are also much more likely to be deterred by issues around the risk and cost of deployment. This perception is likely to be at least partly due to the fact that less advanced users are currently using less sophisticated technology and therefore do not currently have the systems in place to reduce cost (by using open source and by reducing management costs, for example) or reduce risk (by not having to migrate data from one platform to another to crunch it.)

“For any business deterred by a perceived lack of resource and in-depth skills or by the prevailing culture from implementing the Big Data approach, technology can be that enabler,” added de Montcheuil.“Businesses need technologies that enable non Big Data experts to use Big Data; that would allow a regular data integration developer, somebody who has been working with ETL for a few years, for example, to create and deploy Big Data processes. 

“They need to get the right equipment for their Big Data journey that allows them to master the complexity of the environment without necessarily having to learn it all and become deep technical experts,” he continued. “This in turn can help remove the perception of risk and a negative culture around Big Data deployments, potentially giving the confidence to execute commercial deployments. But they need to act fast and start making the move now, if they want to avoid being left behind in an increasingly competitive marketplace.”