Wayback Machine gets giant new datacentre

The Internet Archive organisation plans to announce the opening of a new datacentre to house two petabytes of information for its Wayback Machine, which houses 85 billion Web pages archived for more than a dozen years.

Share

The Internet Archive organisation plans to announce the opening of a new data center to house two petabytes of information for its Wayback Machine, the digital time capsule that stores archived versions of Web pages dating back to 1996.

For example, this is what ComputerworldUK's sister site in the US looked like in 1997 looked like in 1997, what Google looked like in 1998 and what CNN looked like in 20000.

The Wayback Machine houses 85 billion Web pages archived for more than a dozen years, which amounts to three petabytes of data, or about 150 times the content of the Library of Congress. Only five years ago, the Wayback Machine contained about 30 billion Web pages. It is expected to continue to grow by 100TB of data per month now that it's live.

Wayback Machine gets giant new datacentre The Internet Archive's massive database is mirrored to the Bibliotheca Alexandrina, the new Library of Alexandria in Egypt, for disaster recovery purposes.

According to an event invitation from Sun Microsystems, the Internet Archive is moving from a traditional data center filled with standard Linux servers to one that runs Solaris 10 with ZFS on Sun Fire x4500s servers inside a Sun Modular Datacenter. The modular system is an all-in-one data center housed in a metal shipping container for mobility.

Because of the modular design, Sun said the data center was deployed in a tenth the time it would take to build a typical bricks-and-mortar data center.

The Wayback Machine Sun Modular Data Center can service 500 inquiries a second, Sun said. A spokesperson for the Internet Archive said the user interface on the Wayback Machine will not change.

The Internet Archive is a nonprofit organisation located in the Presidio in San Francisco, with data centers in Redwood City and Mountain View, Calif. The archive not only keeps snapshots of Web pages, but software, movies, books, and audio clips.

Users can surf the Wayback Machine by typing in the Web address of a Web site or webpage and then choose from a series of dates that reflect the stored images. The site does not currently support keyword search.

Follow highlights from ComputerworldUK on Twitter