eBay reveals the secrets of grid computing
Online auction site powered by 12,000 servers
By Grid Computing Now | Published 12:00, 26 April 07
“And we also have the search infrastructure, that allows people to actually find stuff, and that’s a massively parallelised scatter/gather application, running on another 2000+ servers. That allows us to meet the latency requirements. Really, we’re using network distributed computing for all the same reasons everyone else is – we couldn’t get the scale any other way, or meet our latency requirements, either.
People expect an instant response from a search when they want to do something. And it brings resilience, too – if you lose one server, you don’t lose the whole service,” he says.
As well as the auction site and the search facility, there is continual development work to improve the eBay customer experience. “The environment we use to build eBay.com is what you would think of as a traditional kind of grid – users are able to submit jobs for building the site. By which I mean compiling Java, compiling C++, optimising and compressing things like HTML, Javascript and XML.
“In the past we used to do that in individual machines and it would take hours. We now do it on a grid of some 300-400 machines and it takes in the region of 30 minutes to build the whole of eBay.
“We’ve done 2.5 million builds since the grid was introduced about two years ago. We roll new code to all of the auction platforms every two weeks, and add 300 new features to the site every quarter,” says Strong.
What is “grid”
Strong is keen to stress that “grid” is a broad term that covers many types of computer system, and that it’s something all companies should be paying attention to.
eBay’s view of grid is that it’s a natural extension of the trend towards distributed computing in the data centre, he says.
“I would argue that almost all enterprises, if they run multi-tiered applications, are already running a prototypical grid, a primordial grid. They’re already running applications that are network distributed, and leveraging the network to scale and achieve resilience and all of those good things. They just typically haven’t recognised that they should be treating the whole of their infrastructure holistically, as opposed to in discrete silos. It’s almost a philosophical thing, about viewing your architecture as a whole.
“The reason we want people to really recognise this is that once you do, you recognise sets of technologies and the problems that come along with it.”
eBay has taken something of a leadership role in grid, almost by default.
“We believe we are, in many ways, on the crest of a wave and everyone else is going to have the same problems we’re having, even if they don’t have them today. Managing these very large systems built, ultimately, on commodity building blocks, means we’re at the extreme edge of what everyone else is beginning to do and where everyone is heading in their infrastructure.
