How Ocado plans to replace its data centre hardware with Kubernetes running on workstation PCs

Computerworld UK speaks to Ocado's head of infrastructure, Chris Dabrowski


Ocado has open sourced Kubermesh, its Kubernetes container cluster software package that enables distributed computing to be carried out on small computing devices such as workstation PCs.

Ocado developed the technology in-house as a means of removing the need for data centres at its huge warehouses. It has also made the code available via GitHub. Instead of running workloads on servers, Kubermesh allows computation to be spread across numerous nodes, such as Intel NUCs or the workstations used by warehouse staff.

ocado dordon warehouse

It is early days for Ocado's own use of the technology – it has completed a proof of concept using Intel's devices – but there are high expectations. "Kubermesh is a mechanism by which we don't require a data centre of any kind," Ocado's head of infrastructure, Chris Dabrowski, tells Computerworld UK.

He adds: "We don't need a data centre in a physical form or the air con it requires, the network that would entail, data centre switches and so on. Kubermesh can distribute the same capacity that was provided by the data centre throughout hundreds of individual high-power workstations which are distributed through the warehouse."

On-premise data centres

Although Ocado is a heavy user of public cloud computing – running much of its analytics and machine learning requirements on Google Cloud – it also has substantial investments in its on-premise data centres. This infrastructure is used for a variety of purposes, such as running its robotics systems in place at its newer facilities. 

See also: Ocado's OpenStack private cloud ensures low-latency connectivity for warehouse robotics

As part of a wide adoption of open source technologies, Ocado is running OpenStack in its new warehouses, automating the running of its infrastructure. More recently, though, the company has deployed containers to run on top of OpenStack or on bare metal systems. It started out with Docker, using FleetD as its container orchestration tool. Now Ocado in the process of moving systems to Kubernetes – the open source container platform which is quickly becoming the standard for container management.

The next step in this process is to create a distributed data centre architecture using Kubernetes.

"Kubermesh allows us to run Kubernetes; both to deploy it and instantiate it across a distributed set of machines," Dabrowski says. "You can look on it as a worm that propagates a beneficial application on top of it.

"Whereas in previous generations we have either FleetD or Kubernetes running over OpenStack, you need to have that environment up and running first, while with Kubermesh you can deploy Kubernetes with nothing running at all."

He adds that Kubermesh runs "in addition to Kubernetes". "It runs beforehand to instantiate it. You run Kubermesh first and Kubernetes is instantiated by that process." 

Resilience and cost benefits

There are a number of advantages to managing infrastructure in this way, says Dabrowski, allowing the data centre hardware to be spread throughout the warehouse in "an organic, dynamic and fault tolerant form".

As well as resilience, cost is a clear factor. Ocado currently relies on pre-built 'container' data centres to support its operational systems. Getting rid of this hardware could bring about a reduction in cost "in the order of seven figures" according to Dabrowski.

"There is a very significant capex cost of building a container computer room and the capex of the network infrastructure is also quite high," he says. "By moving towards a distributed workstation-like server model we don't have either of these costs, or the opex cost of maintenance going forward." 

Future uses

Dabrowski sees the potential to use existing workstation systems to provide the computational power for its operational systems.

"In the warehouses there are many station systems for people to use; fixed stations where people pick items and then move it to a customer bag, but also display stations screens and environmental monitoring solutions. So we thought to ourselves, why not use all of these systems that provide some capacity of compute, make them bigger and distribute the entirety of data centre capacity not within the DC itself, but alongside and internal to these distributed and small-scale systems."

Dabrowski says that while a proof of concept has been put together using Kubermesh, plans to replace its data centres are at an early stage.

"We may use it in future warehouses," he says. "It may not make sense to deploy it into our existing warehouse because we have already invested the capital in the DC infrastructure and the servers themselves. But in future ones this could be an option for us to use to avoid those costs." 

"Recommended For You"

Mesosphere CTO Tobias Knaup on how DC/OS 1.11 takes aim at edge Ocado: Online food is not our bread and butter