Making sense of Big Data

Organisations that manage and analyse Big Data successfully know success begins with a clear business case for an initial product plus a technology infrastructure that can offer scale and management capabilities across private and public cloud. Are you ready for the challenge?


This article is brought to you by ComputerworldUK in association with Intel IT Center

Big Data may be an enterprise IT buzz word, but it is also transforming business. Big Data is all about gaining actionable business insights in real-time, from high volumes of information, or it is nothing. That data is coming from a huge variety of sources, in multiple different formats including social media, mobile web traffic and IoT-generated sensor data.

Organisations that manage and analyse Big Data successfully know it demands a clear business case plus a technology infrastructure that can offer scale and management capabilities across private and public cloud. That means cost-effective and innovative processing and analytics technologies.

Like all IT investments, Big Data projects need to deliver a return on investment - and that means having a clear business case. Some of the most successful Big Data projects have focused on a single achievable goal, such as a new revenue stream; faster time to market for a particular product line; or measurably higher levels of customer engagement. Initial, quantifiable success, can then lead to further benefits.

One example of a business that developed new revenue streams from Big Data is retail fashion analytics company Editd. The business aggregates fashion trend and sales data from a variety of sources around the world and sells them to retail clients.

According to Fortune magazine, Editd’s dataset includes 53 billion data points on the fashion industry, dating back more than four years; covering over 1,000 retailers around the globe. The information comes from retail sites, social media, designer runway reports, and trends blogs.

WGSN is another fashion analytics firm with a dataset of more than a million products and 11 million daily SKUs, from more than 10,000 global online brands and retailers. Like Editd, WGSN makes its money from helping businesses to understand current and future trends.

Another example of this is the song identification app, Shazam. The tool enables record labels to find out where music sub-cultures are arising by monitoring the use of its service. It includes location data from the mobile devices that use the app, so record labels are able to find, and then sign up promising new artists or remarket their existing ones accordingly.

Big Data Requires Data Centre Refresh

In order to capitalise on Big Data, the data centre needs to be sufficiently modern to handle high volumes of fast-moving, diverse data more effectively and data centre management tools need to be able to handle both on premises and cloud solutions.

Server vendors are now producing products targeted on serving Big Data projects, many of them based on the Intel Xeon processor E5 and E7 families, which have been designed to handle Big Data cloud environments. This means they can cope with larger volumes of fast flowing data and workloads, and scale up workflow processing as and when required. In addition, advanced storage capabilities are available through Intel Solid-State Drives (SSDs), featuring high-throughput and high endurance. Additionally, Intel Ethernet 10Gbit Converged Network Adapters provide high-throughput connections for large datasets.

These new technologies point to the broader data centre move to Software Defined Infrastructure (SDI) - whereby data centre operators virtualise hardware assets such as networking components and storage subsystems. This allows them to unify compute, storage and networking resources, and benefit from increased flexibility and scalability, all of which help with handling Big Data.

New Analytics/BI Software Tools

Analyst Forrester Research advises its clients to investigate new data discovery tools, predictive and text analytics, and geospatial technologies – as well as new types of databases and Hadoop-based data stores. Big Data technologies offer the opportunity to close the gap between the data that's available and the ability to turn that data into insight, says Forrester.

The move to Big Data brings with it new types of database. Modern analytics-centric databases include in-memory, columnar, XML, index and associative databases, which can carry a lower maintenance cost, and provide agility and flexibility than older, row-based online transaction processing (OLTP) database management systems (DBMS). In some cases, the newer, more agile database technologies are replacing, rather than augmenting traditional OLTP RDBMS.

Forrester sees Hadoop-based repositories (termed ‘data lakes’) as a key platform element for enterprise data hubs, as part of the agile Big Data hub-and-spoke data architecture. Using Hadoop in this way can bring immediate, and even truly disruptive, benefits, says the analyst.

In future, BI and advanced analytics will become inseparable, with predictive analytics fast becoming another key capability in a broad BI suite. As enterprise BI platforms grow and mature, leading vendors are building predictive analytics organically; or acquiring or integrating third-party predictive analytics platforms. Text analytics and natural language processing (NLP) are also becoming part of this trend.

Organisations are also going to make analytic based insights available to their customers - via embedded BI, which is gaining traction and adding new capabilities. Embedded BI — reports and analytic content integrated into other applications — has been around for a long time. But companies increasingly feel they need to give customers more information about progress of an order or delivery of a service, for example, or deliver that information in a more timely and accessible manner.

A big benefit of embedded BI (versus a standalone BI report or a dashboard) is that it is more relevant and personalised, being delivered in the context of the operational application. This is good news for customers who might want to, for example, interact with their credit card statement, with embedded BI making it possible.

Big Data analytics tools are already powerful and will get more powerful and intelligent. As they do so, enterprises will increasingly be able to handle faster and more complex data. Those that don't take advantage of the new technologies will rapidly lose competiveness.

The impact of Big Data and the ability to deliver real-time, actionable insights, will be felt across industries, as retailers benefit from transportation or weather data; or health and finance companies cross-pollinate their data findings. It’s a brave new world, with almost limitless possibilities.

"Recommended For You"

Craft your future state BI reference architecture Gartner: Hadoop will be in two-thirds of advanced analytics products by 2015