The newly merged company went from zero to a production-ready OpenStack private cloud in just seven months. With the help of vendors Red Hat and Nuage, it created a pilot running a couple of hundred nodes and is now set for full production status this June.
The private cloud will host Betfair’s exchange – a service similar to the stock exchanges – which supports 120 million transactions per day, 2.7 billion API transactions and has 1.7 million active users. It’ll also offer scope for the company to integrate the exchange with its physical stores, as well as powering its casino games that customers tend to play in between betting.
Betfair picked Red Hat's OpenStack distribution because it needed to ensure as little downtime as possible. In addition to Betfair’s core gambling customer base, professional traders also operate on the exchange – so seconds matter and downtime is noticed practically immediately (although Haigh says it’s about 1.5 seconds).
“You can imagine at the end of a horse race, traffic gets fast and you don’t have a lot of time to recover from problems,” Haigh says, speaking with Computerworld UK at the OpenStack Summit in Austin. He adds: “We also wanted to bring support in-house – I wanted to build a team that could look after the estate themselves so I needed partners who were happy to come and train our guys up.”
“We will always have underlying support behind the scenes but I need a team that are my first responders in-house.”
The training for this new platform wasn’t straightforward. But the vendors were responsive.
“The off-the-shelf training from Red Hat just wasn’t ticking our boxes,” Haigh says. “So we had to go back and say: we have to pretty much write our own curriculum with you. So we put together specific training for the Paddy Power Betfair guys based on what we had done and what we were using. That’s been successful.”
The Nuage training, Haigh adds, was tailor-made – because the company was involved from the start.
“What came across very strongly when we were picking from the crowd was that Red Hat and Nuage together understood us, and we understood them,” Haigh says. This line echoes a wider sentiment at the Summit: that technology can be trivial to success when compared to culture.
Betfair Paddypower hopes that by deploying this Openstack system it will simplify the management of its infrastructure. At the same time, the perception of overcomplexity in the ecosystem is not necessarily a bad thing – with the Red Hat system providing plenty of scope for bringing in extra features and diving into those complexities should the company choose to.
Here’s the timeline. Betfair began thinking about the project one year ago. It wanted to match the deployment to the company’s own ethos – pace. Betfair first examined the market to see what was out there, and the proposition from Red Hat for core infrastructure and Nuage for highly agile networking infrastructure proved the winners.
There was one sore sticking point, however: software defined networks.
“As many analysts or consultants we talked to said you should do it or you shouldn’t,” Haigh says. They were split down the middle. “It was very binary.” Although, he adds, at this stage of the project "software defined networking has worked, if I could go back in time there would be no question”. It’s a pleasant surprise.
Once the business had decided OpenStack was the way to go, it took just a week and a half for Betfair to initiate a proof of concept, matched with with performance and functional testing. “That was in part to satisfy, legally, that what everyone said they could do they could actually deliver,” Haigh says.
“In four weeks we delivered it, we proved it. And we were happy to carry on. We then went into the pilot which is just ending now – it was from zero to building up a fit-for-purpose production-ready stack with all the tooling in place.”
In total the entire project took just six months.
“We put our first production node properly on it last week,” Haigh says. “Two of our applications have been running 100 percent moved over for the last week. We’re now in a migration project and that is to take each of the 200 or so applications and to move them. Some of them we have to re-architect a little bit.”
“I reckon we’ve got about 18 months of work to get those applications across. And at the same time new applications will go directly to that new infrastructure.
“I’m excited. It’s been an awesome project, it’s been great working with Nuage, it’s been great working with Red Hat. Particularly the relationship between Nuage and us has been good. It’s been challenging but I think both companies have enjoyed stepping up to the challenge.
“It’s nice to work with fiercely intelligent vendors and fiercely intelligent guys internally, and all come together and everyone benefits from it.
“I can’t imagine we’d have done this without open source at the crux of it – you just wouldn’t get the transparency.”