ITV has migrated its operational analytics workloads away from Comscore and onto the Google Cloud Platform (GCP) stack in a bid to better manage its growing scale, improve its on-demand Hub platform and attract better engineering talent to the broadcaster.
Speaking to Computerworld UK last week, Andy Burnett, director of online technology and operations at ITV said: "Four years ago we realised we didn't have enough data on video quality metrics - excess buffering or failures - so we didn't know if the service was good or bad."
A strategy was put in place to measure "anything that moves" and "collect as much data as possible" but the tools the business was using weren't going to suffice at the scale the service was starting to run at. Especially after a busy summer of World Cup and Love Island fever, peaking with 27 million tuning in to watch England's semi-final loss to Croatia with 1 million streaming live on ITV Hub; and 4 million a day watching Love Island, combining to drive as many as 1 billion data events on a single day.
"Before we ran Comscore, like most broadcasters, so we had to unpick that as a tracking solution for our apps and sites," he said. Comscore relied on heavy-duty SDKs, which were proving sluggish for the business.
"We moved to a spec for tracking data in a uniform way across products," he added. "Engineers loved it for the consistency to implement, and analytics were delighted as they used to spend lots of time ensuring analytics SDKs were right and data was of good quality and accurate, so really we had two delighted internal teams."
Data's coming home
"The requirement to handle the data coming in at the scale it is generated during a World Cup and Love Island is mind blowing and only going to get bigger," Burnett said.
In terms of stack everything is now built around what Burnett calls the "Google Big Data suite" - BigQuery, Pub/Sub and DataFlow. This new stack was assembled in just three months and is known as CROCUS internally.
Where ITV's analytics systems were only able to spot a streaming issue after 10 minutes previously, now, by queueing up usage stats on Pub/Sub, feeding that into Cloud Dataflow and back into Pub/Sub, where it powers a dashboard with metrics, teams can spot issues after 30 seconds.
The team chose Google Cloud as the preferred platform due to its "architecture and engineering decisions". "Another thing that stood out to us at the time was this concept of NoOps, where we don't need to build a 24/7 ops team," he added.
Burnett said that the new leadership of ex-EasyJet CEO Carolyne McCall has led to a greater emphasis being put on data and analytical decision making at the broadcaster.
This includes a more self-service attitude to analytics.
"The requirement in the past was [analysts] want to use a specific tool and we had to build the stack around that," he said. "We decided that was madness, so we would provide rich data to them and they can layer on whatever they need. We output data from Google Cloud and they decide what to do."
Read next: How the BBC rebuilt iPlayer on microservices
The idea is to build an "audience data platform" that will leverage an influx of data science talent at the organisation to better serve those customers. This single view of customer data will allow ITV to "drive better operations" in the advertising business, "improve marketing effectiveness" and "make sure viewers on the Hub get what they want, when they want".
The big red elephant in the room
Of course no conversation about on-demand video can be conducted without mentioning the big red elephant in the room.
"Netflix has changed the paradigm and has done so much to admire," Burnett said. "We are different though and we're not trying to copy them. That is the trick here, to not fall into that trap, so we admire them but will look to tread our own path here.
"Not all viewing is binge-watching drama, there are huge live moments and as that goes online we need special technology to ingest that data and have it operationally, not just to ensure our systems are in rude health but also to then drive new elements like greater personalisation."
Now that this modern data stack is in place ITV is eyieng more advanced analytics. "The first frontier is greater personalisation on the hub to use our own data to drive satisfaction for users and greater consumption, converting them to views as that is a competitive market out there," Burnett said.
Moving towards cloud technology is also proving useful when it comes to recruiting the best engineers. "When trying to attract talent in data science and engineering they are drawn to the new exciting tools and we don't have a huge amount of legacy to deal with," he said. "It's not quite a clean slate, but they have the chance to come in and shape it themselves."
Finally, Burnett said that McCall has earmarked more funding for his department to "do more with data and hire more data science stuff and expand data analytics" across ITV in the coming year. Now we just have to hope Gareth Southgate and Caroline Flack can keep up their ends of the bargain.