In this feature, Dilip Boury cuts through Big Data's buzzword status to give us some practical advice about how to deal with the enormity of it all.

When we talk about Big Data the statistics are incomprehensibly huge. Since 1998, the number of Google searches daily has grown from 9,800 to 5.1bn and there are now 4bn YouTube uploads and 58 million tweets daily. Approximate 11% of the world is now on Facebook and they upload 70 billion new pieces of content every month. This proliferation of data is happening so rapidly that IBM estimates that 2.5 quintillion bytes of data are created daily and that 90% of the world’s data has been created in the last two years alone.

The world of Big Data provides us with both an amazing opportunity and a huge challenge. How can we make sense of all this data without becoming overwhelmed? How can we distil this data into useful information that can be used to help us work smarter? The answer is we need to get better at understanding of how, as human beings, we process data and turn it into useful information.

Cognitive overload

In Mark Haddon’s acclaimed book ‘The Curious Incident of the Dog in the Night Time’ the protagonist, Christopher, is a young boy with Asperger’s syndrome. He says that ‘people never look at everything - they do what is called glancing, which is the same word for bumping off something and carrying on in almost the same direction, e.g. when a snooker ball glances off another snooker ball.’ However, Christopher says he cannot do this because he notices every single detail of every scenario he finds himself in and this leaves him feeling overwhelmed and unable to focus on what he’s doing.  

"We need to get better at understanding of how, as human beings, we process data and turn it into useful information."

Christopher’s experience is a useful analogy for thinking about the challenge of dealing with Big Data. This is because many organisational technologies collect data in much the same way as Christopher’s brain; they automatically record every single detail of every interaction. For example, call centre technology collects statistics about every call, supermarket databases store point of sale data on every purchase and manufacturing technologies record the precise time and cost of each stage of their production process. These systems create more data than we can possibly process and which can leave us unable to see the wood for the trees.

Getting a grip on Big Data

• Keep a narrow focus

It is important to only collect data that you know will be useful because otherwise you may become distracted by spurious information or so focused on detail that you lose sight of the bigger picture. For example, lots of organisational surveys ask far too many questions and report the data in so much detail that nobody could ever hope to respond to the issues they raise. This means that it is not obvious what action to take and therefore little progress is made. These organisations would be better off identifying what information would be useful to increase their performance and to focus their attention here.

• Cognitive offloading

Another way of ensuring you are able to take advantage of Big Data is to reduce the amount of unnecessary information processing your brain has to do, so you can focus your attention where it is most needed. One useful way of doing this is by offloading some of your memory and information processing onto your environment. For example, lots of information is at our fingertips through smartphones. Social networks, wikis and cloud technology mean that information can be accessed from anywhere. All of this means that it is now far more important to know what data sources are available and how to access them than it is to be able to memorise and recite long lists of facts by rote.

• Affordance

If the way data is presented suggests or affords a particular course of action then people are much more likely to be influenced by it. Consequently, we can reduce the cognitive load placed on people by Big Data, by making it easier for people to understand and interpret what it is telling them. This can be done in a variety of ways, e.g. graphically representing data, pulling out and summarising the key messages, or by using database technology to make it interactive. The key to success with this strategy is to be clear about what you are asking people to do and to present the data in a way that aligns with this goal.  

• Situated cognition

The cognitive load placed on people by big data is mediated by the situation in question. For example, a CEO of a global company may take a look at the productivity results for the last year to get a sense of whether their organisation has the right strategy and direction; however they are unlikely to have the time to drill down into the detail for each department or role. Conversely, a frontline manager is less likely to look at the big picture and more likely to focus in on the specific issues in their department over the last week or month. This demonstrates that there should not be a ‘one size fits all’ approach to Big Data. Different people will have different needs and they will require tailored support to understand and make the most of the information they are presented with.

• Call in the experts

With numerous heterogeneous data sources available in organisations it can be difficult for leaders and managers to draw it all together into one big picture. With Human Resources data, for example, different data sources may give different messages and there may be no simple way to ascertain which is correct. At this point it may be beneficial to bring in someone who is a specialist in business psychology or workforce science. Part of their academic training is to understand how reliable different data sources are and what inferences can be validly drawn. These specialists can help you recognise the key messages and strategies you may need to put in place to deal with them.  

Dilip Boury is a smarter workforce consultant at Kenexa, an IBM company