Inside AT&T's secret disaster recovery centre

Somewhere in an unmarked warehouse in England, modular trailers packed with data centres and networking equipment whir and hum. This is where global telecommunications giant AT&T keeps its disaster recovery equipment, ready to deploy in the wake of tsunamis, earthquakes and ice storms.


Network operators know the usual pitfalls: cut cables, software outages, power cuts. Here at an AT&T warehouse at a secret location, the company also keeps everything that’s required to restore connectivity to crisis-riven parts of the world, from geopolitical groundswells to earthquakes, tsunamis, ice storms, and even terrorist attacks.

The facility itself, one of four in the world, is humdrum – a nondescript exterior and the plainly industrial appearance of its largely empty, grey interior belying the thousands of miles the equipment housed within will travel to, and the gravity of those situations.


This warehouse is geographically important because it is relatively easy to deploy equipment to anywhere in the world thanks to its proximity to global freight hubs in the UK and Europe – and in case of disaster at the Rio Olympics, it would be shipped out from here.

When a 8.8-magnitude earthquake hit Chile in 2010, Justin Williams, director of the Network Disaster Recovery group for AT&T, headed a team to restore connectivity to the region. It was live within the week.

AT&T deployed the same gear that a small group of us are shown in the warehouse: trailers fitted with their own batteries and generators, reams of fiberoptic and copper cabling, network equipment placed in bays and racks that can scale up to a capacity of more than 15 terabits per second.

“We were on the first non-humanitarian flight,” Williams says, referring to his team and 40 tonnes of equipment. “In a situation like an earthquake, basics become really critical. When we arrived, nearly all of the hire cars were gone – there’s a lesson learned there – we understood we took our own power, our own generation systems, but fuel, of course, is another thing.”

“A learning point from Santiago was trying to get hold of fuel. Now, we take all these empty jerrycans with us – so we can go off hunting for fuel if necessary. You’re not going to find these in a local hardware store when there’s been a disaster.”

“Self-sufficiency is the key,” explains Williams. “You can’t take a team of people and immediately create a burden somewhere else. You need to have as much as you can with you and people who are trained to look after themselves.”

Situated by the trailers are ‘bugout bags’ for the volunteers to take on their deployments, including equipment like a Broadband Global Area Network (BGAN) satellite system, which can be placed anywhere in the world and configured to create an emergency wireless network.

“Everything is measured and weighed, so when we deploy it, we know exactly how much everything is going to weigh,” Williams explains, who is in charge of most of the designs. All of the trailers are in something called ‘MGX containers’, vital for flying equipment overseas.

“The design fits into a particular slot of a Boeing 747,” he says. “If it had a more chamfered edge on the front, it could fit anywhere on a 777. This is certified for air travel which means all the air crew, the ground crew, know how to handle it.”

More than 100 AT&T staffers worldwide volunteer as part of the NDR team. There’s a separate International External Affairs team that assists in procuring travel visas through a fast-track process, plus a medical team to keep track of inoculations for conditions like typhoid and tetanus. Then there is also an operations team that deals with day-to-day practical logistics: accommodation, supplies, travel, etc.

All of this is supported by the GNOC – the ‘Global Network Operations Center’ – which tracks global affairs, anything from political groundswells to natural disasters, to prepare the NDR group for the possibility of outages.

Williams wouldn’t tell me if there were any events the company thought would lead to disaster but then didn’t, but he assured me the GNOC keeps a close watch on everything. For the operations of a telecommunications company, it’s all rather reminiscent of cloak and dagger cold war-era spooks. Speaking of which, Williams confirms that AT&T personnel are trained in restoring networks in the event of nuclear disaster, from an attack or otherwise.

“Our special operations team is trained to work in chemical nuclear and biological suits,” he says. “That would be a response to the aftermath. We have a personnel response for it.”

Concepcion, Chile, 2010. Image: European Commission/Flickr

According to Williams, competitors have “some capabilities” but AT&T’s are relatively unique in the industry. The company has invested more than $600 million (£454m) to date.

“It’s not something we charge the customers for, it’s something we do for ourselves – we are recovering the network on behalf of the customers,” Williams says. “Once you step through the door, and I dare say no one ever thought it was going to reach $600 million, it starts to grow. Once you buy two trailers, you need some method to move them and the processes and procedures to do that. You grow from 10 to 20 to 200 vehicles and your capability increases within it.

“You need to invest in recovery operations, training, educating a larger workforce. You’re spending a lot on warehousing and maintenance – air conditioning units, HVAC units, all the generators need maintaining. Once you commit to this kind of capability you’re really opening up a business unit in its own right, that has to be kept in top condition. These are all the different elements not even mentioning the technology at the heart of this – you have got to keep reinvesting.

“But despite the changes in direction at AT&T, or the changes at the middle or senior executive levels, we keep invested, we see the value, and at no point has anybody said: ‘we should stop this.’”

“We know that for a significant, catastrophic failure, this is really the only game in town.”

"Recommended For You"

How Cisco's TacOps unit restores communication after natural disasters Dealing with key trends during an IT disaster recovery