Big Data is a big deal in many organisations. Few business unit managers would not be excited about the opportunities that near real-time business intelligence offers their department.

The prospect of on-the-fly updates to meet changing customer needs, as well as deeper insights that can be gleaned from the mountains of data organisations hold is compelling.

Most CIOs will be keen to embrace the Big Data concept, and argue for investment in the new servers, networks, storage and management tools it requires. IT security staff, on the other hand, have tended to react more cautiously.

Perhaps caution goes with the job description, but perhaps IT security specialists should see Big Data as a major opportunity to beef up their defences.

Caution is understandable. Big Data means increased exposure of company data, particularly if it is stored in a single warehouse. But Big Data technology and techniques can considerably enhance IT security professionals’ hunt for threats and systemic weaknesses.

Analysing network behaviour has been well established as a means of identifying anomalous activity, which could be a sign of a hack or virus. But for too long network managers and security professionals have complained about drowning in the volumes of data thrown up by network logs. For too long they’ve been overwhelmed by false positives thrown up by security systems.

Now Big Data techniques offer a way to make sense of these logs, to eliminate the false positives and act more quickly on the real issues.

But this is just the beginning. With Big Data the hunt for anomalous behaviour can be extended to the entire IT estate. It can be used to model ‘what-if’ scenarios that businesses can use to help understand how changes to their organisational governance or security systems might affect their vulnerability to attack.

Such systems may take data from networks logs, application performance and operating system monitoring. They could also include data capturing employee and customer behaviour as well as details about IT systems.

The potential for these tools is that they could allow business to ‘try-out’ changes to security policy or technologies. Detailed, realistic modelling without having to go through the effort and cost of doing it for real, and finding it does not work and repeating the process, is now possible. Organisations won’t have to go through a virus or hacking attack, or perform their own stress test, to figure out whether they have chosen the right approach.

Big Data can be part of the solution, not part of the problem. However, as ever with IT security, there is a caveat. The Big Data-style security analytics is only as good as the data you put into it. Dirty data means inaccurate forecasting and lost benefits.