The exponential growth of data, ever stiffer regulatory requirements, the rising importance of green issues and the recession pose unique problems for IT professionals, but there are technical solutions at hand.

Levels of data within organisations are growing exponentially. According to Nemertes Research, businesses’ data storage space requirements are expanding by up to 150% each year.

Not only is there more business critical information to process every day, but there is the heavy burden of regulation and compliance to consider too. So, with data centres already bursting at the seams, and a wider pressure to consider the environmental impact of IT, companies are faced with a significant challenge.

This challenge requires an absolutely fundamental change in the way that companies operate and run their business. Data needs to be extremely well managed and analysed within all organisations, but in an economic downturn it has never been more important to consider the resulting impact on profitability and efficiency.

Yet, many corporations are having to juggle ever-changing regulatory demands with increasing requirements for fast access to and analysis of data, along with an added pressure to manage carbon footprints. The result can be a data management and analytics minefield.

The Challenges

Regulatory compliance

Sarbanes Oxley (SOX) and MiFID, along with other compliance initiatives, bear great pressure on companies. Not only do they demand the storage of vast amounts of data, but also that it may be accessed on-demand.

On top of this, the requirement to monitor unstructured data such as emails, instant messaging (IM) conversations, voice recordings and most recently, social networking communications adds further complexities. The government is placing companies under ever tighter controls around the volume and transparency of data. Ignorance is no longer an excuse for failure to comply.

One of the industries hardest hit by the changing requirements is telecommunications. Stringent anti-terrorism laws means that companies in this sector must keep records of all voice and data traffic over long periods of time; from where a call was made, to whom it was made and occurrences of key words.

After the London bombings, telecommunication companies had to access and analyse huge amounts of traffic to help the government and comply with its regulations. Tasks such as these are mammoth and if data is not organised appropriately, it can soon become mission impossible.

As if this was not difficult enough, there is a rapidly increasing obligation under the Carbon Reduction Commitment to drive down energy usage or pay the financial penalty. This means more regulation but acting completely contrary to the data capture requirement squeezing business from both sides.

Fast Access to Data

Effective Business Intelligence (BI) allows companies to make better, more timely business decisions; a critical success factor in the modern business landscape. By reporting on, analysing and optimising business operations, costs are reduced and revenues increased. This may seem straightforward, but to do this accurately, all data that is stored must be accessible for analysis in near or real-time.

Real intelligence depends on closely targeted real-time queries that provide detailed information. A call centre representative might use a dashboard that creates a single view of a customer, assembled on-the-fly from a variety of disparate data sources.

In such cases, speedy access to information is vital. BI users are often on the front line and simply cannot wait for data to be staged and loaded or for long-running query results. The expectations of modern customer service mean that they need the information immediately.

The challenge for organisations is that data volumes and query complexities are growing rapidly. Businesses need answers fast, regardless of where the source data is housed. And, with compliance reporting pressures increasing, it is unlikely that sources will be confined to traditional data storage devices alone.

The bottom line is that for BI to work, users must be able to pull data from multiple sources to get answers in real time, no matter where those users reside in the flow of business information.

Storage, and the effect on carbon emissions

With compliance pressures and the need for real-time access to data comes the challenge of data storage and an increased carbon footprint.

This requirement is exacerbated by the fact that to make this data usable actually increases the volume of storage over and above the raw data requirement – anywhere between three and sometimes ten times the original requirement – with a corresponding impact on power usage and emissions.

As companies employ more servers to accommodate the burgeoning data levels, power and cooling demands increase considerably. This has an inevitable knock-on effect on energy consumption and therefore CO2 emissions. In fact, Gartner claims that the IT industry is now responsible for as much carbon output as the aviation industry.

Companies not only need to address the challenge of storing additional data, but also to reduce storage consumption; especially if they are to avoid increasing external legislative control.

Facing the Challenges

For once, technology must drive the business solution rather than the other way round. By investing in the correct technology, businesses can avoid a situation where critical information cannot be retrieved and actioned.

Yet, too many fail to understand how this investment allows them to meet compliance requirements, achieve fast access to data, save money and reduce carbon emissions.

Businesses need to stop throwing money at expensive servers. These hardware vendors can claim to address some, but not all of these challenges. In fact, hardware alone will increase data centre storage costs.

For example, by investing in hardware solutions to lower cooling requirements and enhance data capabilities, companies are only solving the problem of access to data, whilst increasing data centre storage costs and carbon emissions.

The solution is to revisit the software within the data centre. By looking to software, rather than hardware, companies can compress data and reduce storage requirements, whilst making the required connections and managing data with the required industrial strength.

Such a change in approach does three things. Firstly, it compresses the data significantly to reduce overall storage requirements and the associated impact on carbon emissions through reduced cooling and power consumption.

Secondly, with data taking up significantly less space, the throughput of queries is increased; allowing information to be accessed and utilised much more quickly. Finally, this fast access to data means that businesses are extremely well-placed to meeting all regulatory compliance requirements.

Implementing more power-efficient analytics software is the solution that will allow companies to manage explosive data growth efficiently without the addition of hardware that’s not only expensive to purchase, but to maintain as well.