The battle to make sense of the data explosion

In only three years, the bytes of data generated by digital cameras, mobile phones, businesses IT systems and devices will equal the number of grains of sand on the world's beaches.

Share

In only three years, the bytes of data generated by digital cameras, mobile phones, businesses IT systems and devices will equal the number of grains of sand on the world's beaches.

It's a mind-boggling estimation from market analysis company IDC. But it reflects the proliferation of devices and systems used by consumers and businesses, said Stephen Minton, IDC's vice president of worldwide IT markets and strategies, during a briefing yesterday.

Over the next few years, corporations will face tough decisions on how to store data, find information and comply with regulations, said Minton, who laid out a forecast of what IT managers can expect from what's being labeled the "information explosion".

It won't be an easy task. While 85 percent of that data is predicted to come from consumers snapping photos, surfing web pages and sending email, about 60 percent of that consumer data will still cross corporate networks, Minton said.

Much of the data is unstructured, meaning it's not clearly labelled as to its content, such as photos, video and perhaps phone recordings, which makes it more difficult to use.

But technologies that enable deep analysis of the data are emerging, and could help business unlock what's important and improve their operations.

The mass of data companies are collecting - the US retailer Wal-Mart Stores for example, generates 1TB of transactional data a day - represents huge business opportunities "if companies can really get to a point where they can successfully analyse the data going back and forth on their network", Minton said.

Companies are "still a long way short of being able to analyse the unstructured data on their networks", he said. The data is coming from web services such as the photo-sharing site Flickr, Skype and even the virtual world Second Life.

The business opportunities are the positive side. But the security concerns still abound, as well as regulatory compliance and liability worries.

Find your next job with computerworld UK jobs