The world today is being flooded with digital data, in myriad manifestations and washing over us at such incredible speed that making sense of it is dauntingly difficult. Yet this tidal wave of data, when channelled and filtered by an array of new technologies, holds untold value for organisations. Or so we are told. But despite the sometimes exaggerated hype surrounding "big data," the fundamental assertion is true: data and the decisions driven by that data now represent the next frontier of innovation and productivity.
Big promises to fulfil
But for most businesses, the promise of big data is nowhere close to being fulfilled. For one thing, spending on it is polarised. While the telecommunications, travel, retail, life sciences, and financial services industries are making significant strides in big data technologies, other industries, such as manufacturing and government, are taking a wait-and-see approach.
The disparity between a few success stories and the lack of action elsewhere has created a high level of anxiety within firms that have not yet begun to explore big data. But it is important that they not rush thoughtlessly into the fray. An organisation should make a big data investment only if it has well-defined and realisable business objectives. In the first article in this two-part series, we examine the barriers that must be overcome before businesses can realise the benefits of big data's promise.
Volume, velocity, and variety
Big data is often said to be characterised by 3 Vs: its tremendous volume, the velocity at which it needs to be processed, and the variety of data types it encompasses. The first two characteristics are fairly obvious: technology has made it possible to capture increasingly large amounts of information and make it available for analysis in real time. But mining the value of big data also is difficult because it requires simultaneously analysing various types of information: transactions, log data, mail documents, social media interactions, machine data, geospatial data, video and audio data, to name just a few; much of which is "unstructured."
Traditional types of business data were available in a format that was structured and could have been automatically analysed; for example a spreadsheet quantifying customer returns of different products at different stores over time. However, much of the value in big data exists in unstructured information; such as the transcript of a conversation between a retail customer and a customer service representative. Synthesising unstructured data from numerous sources and extracting relevant information from it can be as much art as science.
Much has been said and published about the looming talent gap in the big data industry. Estimates<http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation> suggest that the United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills, as well as 1.5 million analysts and managers to analyse big data and make decisions based on those findings. There are further predictions<http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation> that only one-third of 4.4 million big data jobs created by 2015 will be filled. Unlike traditional analytics, mining big data requires an extremely diverse set of skills: deep business insights, data visualisation, statistics, machine learning, and computer programming. Policy should work to mitigate this talent shortage through forward-looking education and immigration policies.
Flawed data governance
Big data is not a substitute for; much less a solution for; flawed information management practices. If anything, it requires much more rigorous data governance structures. Without those improvements, IT systems that have not been upgraded to handle large volumes of data are likely to collapse under the sheer weight of the data being processed. Business leaders are often more excited about the potential of big data than their IT counterparts. That may be because of IT executives' understanding of the realities on the ground.
Lack of a data-driven mind-set
Because mind-set can be hard to pin down, its power is often underestimated. That is a mistake when it comes to assessing the prerequisites to successful analytics deployment. It is virtually impossible for big data investments to deliver value if business leaders do not have a data-driven mind-set. That is, if they do not believe that it is important for decisions to be based on cold, hard numbers rather than gut feel and experience. But once the right mind-set takes hold, other good things will follow: data-driven business leaders will have a tremendous incentive to treat data, and therefore the IT and analytics professionals who help deliver it in an understandable form, as a strategic asset. And these leaders will make it a priority to ease the flow of data across organisational silos.
Lack of technical know-how
Big data represents a convergence of IT and data science. Technologies include; Hadoop, which enables large-scale processing of diverse datasets; R, a programming language for statistics; and in-memory databases, where data resides on main memory as opposed to disk storage. Data science includes, among many other areas; systems that learn from data, which has become known as machine learning; and data warehousing. Big data professionals are expected to be familiar with both disciplines, but this combination is rare, despite the training courses that are sprouting up globally.
Having identified the key challenges that are holding businesses back from realising the promise and full potential of big data; the second article in this two-part series will provide nine steps that companies can take to begin turning big data talk into action, and buzz into business benefits.
Posted by By Anant Gupta, CEO, HCL Technologies
This is an extract from Anant Gupta's article "Making Big Data Something More than the Next Big Thing," featured in WEF's Global Information Technology Report'.