ITV has embraced devops across its organisation to support development, enabling the broadcaster to create digital products more rapidly and with greater reliability.
The devops work centres around the creation of what ITV calls its Common Platform. This is a "standardised coding toolkit that gives teams across [ITV’s Technology division] a common way of doing things like: logging, monitoring, alerting, deployment pipelines, configuration management".
Starting off as a pilot project to create a version of its ITV Player online service for connected TVs three years ago, the devops team has since grown from three to 15 engineers, with 16 products now reliant on the shared Common Platform.
The benefits of the devops approach are clear, according to Tom Clark, head of ITV's Common Platform.
"We are doing more changes in parallel than ever before – VMs in minutes not weeks, and greenfield hosting environments in days not months," Clark said, speaking at Cloud Expo in London this week. "I have one engineer who started one Monday and by the time he joined the product team the following week, he had built an entire hosting platform and he literally started and had never seen that tooling before."
He added: "Performance has improved with more eyes on a problem, and reliability is up as well."
From cost-cutting to modernisation
This was not always the situation for the company. Back in 2010, ITV was in financial trouble, and a new CEO was brought in to cut costs. "One of the things he did was to outsource our infrastructure to a managed service provider, which saved a lot of money and helped save the company," said Clark.
The outsourcing arrangement had drawbacks, however, and meant that virtual machines took "weeks" to provision. "That was 2010: terrible times," said Clark.
ITV initially began adopting a devops approach in 2014 to support a project to create software for connected TVs.
"When we did the big cost-cutting we pressed pause on all of our internal development," said Clark. "After a while we realised that actually we were slipping behind the competition. They had cool new fancy systems and ours kind of stopped five years ago. So we did a modernisation programme."
The project, led by another member of ITV’s IT team, served to highlight the barriers the company faced in order to operate at speed.
"He got some devs and he got some ops, and lumped together," Clark said. "He requested some VMs and he waited and waited and he waited some more, and eventually he got so annoyed with waiting for these VMs from the MSP he went on to Amazon, got his credit card out and built the entire stack in about four weeks."
Automation and standardisation
While the developers had been moving from waterfall-based development to agile development, the delivery of infrastructure wasn’t fast enough. "We still had a very waterfall infrastructure, and we realised it wasn't really going to work," Clark explained.
ITV grew its Common Platform approach to address this. It was built on a few key principles, and one is the use of automation tools.
"We want to automate the boring and repetitive stuff, take the humans out of the equation, because when you make smart humans do boring repetitive stuff, they get bored and they leave or they make mistakes," Clark said. "It also means that they can concentrate on the interesting fun work that makes them happy."
He added: "The great thing about automation is that the more you automate, you have more time, which means you have more time to automate, which means you have more time. So you get this virtuous infinite loop."
Standardisation was also key to greater efficiency.
"Standardisation allows you to make assumptions," he said. "If you all agree that you are going to make square pegs of a certain size, when someone else makes a square peg you know can go into their system and take their square peg and fit it into your square hole because you have all agreed on that standard.
"What that means is you have multiple teams developing stuff, you can cherry pick from all of their systems and you basically have all of their work for free. It's kind of like a rising tide lifts all boats."
Since the introduction of the devops approach, the methodology has gradually become more deeply embedded in the IT team. Alongside a 'core' team of engineers - the "first responders" who are responsible for maintaining the common platform, as well as other areas such as R&D – there are engineers in each of the five product development teams.
There are also plans to extend the Common Platform across the rest of the IT estate at ITV, specifically its Windows-based systems.
Overall, Clark said there have been benefits for staff and for the wider business. "The result: happy people, happy engineers, happy devs, happy products owners, happy directors, users and customers," he said.