Market research and data analysis company Kantar Group has begun a major two-year project to consolidate its UK and US data centres and move the majority of its applications into the public cloud.

Kantar Group comprises of 13 specialist research companies delivering customer insight to many Fortune 500 clients. The group, owned by advertising and marketing firm WPP, operates across over 100 countries with a staff count of over 28,000. Kantar has a turnover of around $4 billion globally.

In order to support the various data-intensive research companies that operate under Kantar’s control, the group employs a shared-services IT model, with centralised infrastructure across a number of regions it operates, and managed by 360 IT staff. The centralised infrastructure previously consisted of 9,000 servers across nine data centres in the US and UK, part of a wider network of 37 data centres worldwide.

In 2012 the company began a major project to consolidate its data centre footprint in the US and UK, two of its most active regions, with a view to moving much of the third party and in-house built application workloads, such as finance, costing and enterprise resource planning systems, as well as research platforms, into the cloud.  

What we were trying to do is get out of the physical data centre and infrastructure business,” Kantar Group CIO, Matt Graham-Hyde, explains to ComputerworldUK. “The idea is to take the nine data centres we have got, and end up with a small one in the UK, and a small one in the US. We are pushing everything to the cloud – whether Amazon Web Services, Microsoft or Google.”

Drive for innovation

There were a number of reasons for the migration project. One is cost, with the company expecting to make savings of around 50 percent in IT spend by reducing its data centre footprint. For some applications the cost of running is reduced by up to 70 percent.

Furthermore, the ability to innovate in a marketplace that is seeing start-up research companies launch new cloud-based services has become vital for the company. However, Graham-Hyde said that the huge outlays on physical tin and proprietary databases technology previoulsy made by the firm meant that it was becoming difficult for Kantar to react to new opportunities and compete with start-ups.

By moving as much of its IT estate as possible into the cloud, Kantar would be able to scale its operations much more quickly where needed, and at a fraction of the cost.

We were trying to find a way of trying to reinvent the business model. We want to do that because we need to develop new products and services very quickly. We need to deal with that challenge and we were never going to be able to do that with our previous IT model," says Graham-Hyde.

While Kantar's operations have always been data intensive as a market research business, it is also now needs to deal with an influx of unstructured data that it monitors from social media feeds for example, as well as from a number of new sources.

Big data challenge

Overall the company currently stores 4.2 petabytes of data in its US and UK, with just over 10Pb worldwide. This is set to continue to triple in the next two or three years as Kantar analyses social media feeds and gathers intelligence from other new sources, such as car sensors.

It is growing hugely, and that is just the data that we keep, not the transient data,” says Graham-Hyde. “In terms of the data that we will be processing it is beyond all comprehension, because we have to deal with things like the Yammer feed and Facebook feeds, we have to deal with companies that are rolling out sensors into all of their products. All of this is relevant to what we do as a business, and is reinventing the research space."

The quantities of data that we are dealing with are increasingly going towards ‘huge data’, not ‘big data’, and what we are dealing with will probably triple over the next few years too, so using things like Elastic Hadoop (EMR) is game-changing.”

Kantar Group is now using a wide range of AWS tools, from the relatively well known compute (EC2) and storage (S3) offerings, to a variety of newer and more advanced services, including RedShift data warehousing, Elastic Map Reduce (EMR) Hadoop-based big data crunching tools, and various others.

While AWS will be used for PHP and open source Java applications, Microsoft Azure is being looked to for .NET frameworks. Meanwhile, Kantar Group will use Google Apps Engine and other Google cloud tools for data analysis, Graham Hyde says. There will be a number of software as a service offerings deployed as well, including IBM’s SPSS Dimensions data collections tools.

Migration to AWS

He said that one year into the project, the company has already made good progress with the UK application migration process, working with vendors and cloud providers, and is already running many applications on AWS.  

We have moved applications where we have just lifted and shifted them, and we have moved them where we have converted them to be elastic. We have pretty much deployed all dev/test self-service into Amazon now so the development teams no longer have to call the infrastructure function to set up stuff.”

Although some subsidiary companies were reticent to move to a cloud model at first, the migration is now in full swing, and many security and regulatory requirements have been assuaged.

"You have to create a culture in where they are not threatened by what is happening, and they see it as an opportunity.  Some people will not want this change when you first move it into your organisation, and some people will go out of their way to make this not work."

"The fact is, you are more secure in the Amazon cloud than in your own data centre," Graham-Hyde says.

It is expected that the migration of applications to the cloud will be completed in 2014.

We are quite a long way along in the UK. We are comfortable that we will be finished in the UK by the end of next year. We are not as advanced in the US, but I see it as a snowball effect - as the skills build, as the confidence builds, so does the speed. So I would expect to see the American data centres merge even more quickly," says Graham-Hyde.