Big Data technologies could become 'must-have', rather than 'nice-to-have', tools for certain businesses, in light of the government's plans to introduce legislation that will reportedly give the police access to communications data in real time.

The communications data includes information such as what websites individuals have visited, and who they emailed, but not the actual content of exchanges. This aspect is not expected to change under the new law, which aims to allow intelligence officers to continue to gather data they need for criminal investigations from new technologies such as Skype.

"We need to take action to maintain the continued availability of communications data as technology changes," said a Home Office spokesperson, who said that a warrant would still be required to obtain the data under the new law.

While civil liberties campaigners have been up in arms over the plans, it is not the privacy issues that are new here, but rather the 'real-time access' to data.

Existing data regulations in the UK – such as the Data Retention Directive – already allow intelligence officers to access historic communications data held by ISPs (Internet Service Providers), telecommunication companies and social media platforms, while the Regulation of Investigatory Powers Act 2000 (RIPA) allows access to real-time communications data on a one-off basis.

Although a draft version of the new legislation is not expected to be unveiled until the Queen's Speech in May, the availability of technology to enable businesses to comply with the law has been called into question.


"What's new is that this legislation means it could be demanded in real-time, rather than simply asking for historic data. Whether this is feasible or not depends on whether there is the technology available to deliver that information quickly – otherwise ISPs will struggle to comply through no fault of their own," said Kathryn Wynn, senior associate at law firm Pinsent Masons.

"Given the enormous number of transactions involved there is also the danger that GCHQ will suffer from information overload, trying to sift through millions of calls, emails, texts and web visits to find what they are looking for in real time, rather than analysing patterns in historic data."

Technology cost burden

Wynn believes that there will be high costs involved for ISPs and telcos unless technology that is sophisticated enough to identify and filter out criminal activity in the communications data is readily available.

"With other situations where technology has not been available, there has been a cost burden," she said.

She provided the example of the EU cookies law, which came into force last year. It requires organisations running websites to give users the opportunity to give prior consent to cookies.

The problem arises where browser technology is not sophisticated enough to give users the option to differentiate between useful cookies – such as the ones used by Amazon to remember customer preferences – and 'bad' cookies – where information collected is sold on to third-party advertisers.

"It is all or nothing, and organisations have to develop [the sophisticated technology] themselves," said Wynn.

She added: "It means that for the Information Commissioner's Office (ICO), it has to be quite careful about enforcement. A lot of organisations want to comply, but need to work out how to do it.

"If ISPs don't have the technology that allows that real-time access, they could have that non-compliance."

Big Data

But perhaps the new legislation will open the door for Big Data technologies, suggested Alys Woodward, an analyst at IDC.

"It is doable to look at [this data] in real time – with Big Data architecture – but it is a big challenge. There is almost certainly going to be pushback [from ISPs] saying 'how much is this going to cost us?'" she said.

Big Data refers to the ever-increasing quantity of unstructured data generated by websites, social media and other sources, and the technologies being developed to gain insight from it. While still in their infancy, vendors (such as EMC through its Greenplum division) are developing more and more of these technologies, and a growing number of real-life use cases have also started to emerge.

Forrester recently said that clients are saying that Big Data is now an imperative, not just an initiative, citing examples from the City of Dublin, which is analysing huge amounts of transportation data to optimise public transit, and Seton Healthcare, which is mining unstructured data to get a more complete view of patients.

But the technology is not easy to get right, if the Massachusetts Technology Leadership Council’s Big Data Disruption event held earlier this year is anything to go by.

Speakers at the event said that Hadoop, one of the technologies most closely associated with Big Data, had “terrible performance on data management”, for example. Hadoop is an open source programming framework that allows users to split up large processing jobs and run them in parallel across clusters of servers.

However, if companies are required to invest in such technologies themselves, Woodward believes that they might take the opportunity to use the communications data gathered for their own business purposes. Mobile operator O2 analyses customer data to identify potential customer churn, for example.

"They [telcos] already do this with location-based services. We would see more of that," she said.

Time for everyone to review their data policies?

While the government is working out the details of the new legislation, the announcement should still be a prompt to all businesses to review their existing data and privacy policies, Woodward said.

"[Announcements like this] will always lead some organisations to tighten up their privacy policies. Organisations should consider what they are doing with customer information – are they selling the information to all and sundry?" she said.

"Organisations could be forced to think 'we need to watch our reputation. We don't want to be in the press as people being careless with our mailing lists'."

Pat Phillips, practice director at IT and business change professional services firm Xceed, agreed: "End users should be able to not only ask vendors where their data is being stored, but also get guarantees about its use."