Fast-growing food delivery startup Deliveroo has shifted its data warehouse from Amazon Redshift to new vendor Snowflake in a bid to free up analysts and data engineers' time.
Henry Crawford has headed up the business intelligence team at Deliveroo for nearly two years now, and he told Computerworld UK that his aim is to "make Deliveroo a data-driven company and ensure that data decisions are baked into everything we do here".
As Deliveroo grew the number of orders delivered globally by 650 percent in 2016, and Crawford's team scaled up to 16 specialist business intelligence staff and 35 analytics specialists, the organisation started to feel the strain on its existing RedShift data warehouse.
The old data warehouse "couldn't handle concurrent users", especially on Mondays when bottlenecks would emerge as everyone would be assessing the previous week's performance and running queries.
So Crawford assessed various options for a more scalable data store. He quickly landed on a new entrant to the market, Snowflake, after it proved itself able to handle the data loads required during a 24-hour technical trial.
Deliveroo migrated 7 terabytes of data from RedShift to Snowflake in May this year and Crawford said it was a relatively pain free migration as both sit on Amazon Web Services (AWS) infrastructure.
The company has already scaled this up to 35TBs of data and Snowflake now underpins all of the analytics at Deliveroo, the majority of which is consumed using a Looker front-end.
Analytics at Deliveroo
Analytics at Deliveroo ranges from traditional reporting and dashboards to the machine learning team working with more real-time operational data.
Snowflake now underpins 'Frank', Deliveroo's driver dispatch engine. This machine learning system constantly calculates and recalculates the best combination of riders to orders, crunching historic data for travel and food preparation times, with real-time traffic and weather data.
Crawford says his team is now able to run multiple concurrent queries, with data available in seconds and without having to enlist the help of a data engineer, freeing up their time to help with more value-add projects like the Frank system. In short: "Having a pipeline that runs quickly and efficiently allows us to focus on new projects," Crawford said.
It also allows analysts to focus on their main job, rather than operational concerns like where a query is in a queue. "Simply allowing them to do what they should be doing, which is understanding the business and helping people use the data," he said.
Crawford hopes that the new data warehouse will help the business with its latest initiative, Deliveroo Editions, where the company is building clusters of pop-up kitchens to help restaurants launch delivery services in new areas where there is proven demand.
Analysts are being tasked with using rich data, like local demographics, combined with Deliveroo data to spot areas where there is unmet demand for certain restaurants or cuisines.
"This is a nice example of digging into data and being more flexible on new projects," Crawford explained.