France Télécom's French mobile network, Orange, crashed last Friday afternoon, leaving many of its 26 million subscribers unable to make and receive calls until early Saturday morning local time. The company has promised to compensate customers with free calls, text messages or data for a day, and a ticket to the movies.
From around 3 pm, customers began to lose their connection to the network because of a problem with the Home Location Register (NG-HLR), the database that keeps track of the cell to which each subscriber is connected. The root cause of the problem is still unknown, and Orange is working with its equipment suppliers Ericsson and Alcatel-Lucent to identify it, France Télécom CEO Stéphane Richard said at a news conference Saturday afternoon.
Richard ruled out a cyber-attack as a possible cause, adding that the problem was also unrelated to Orange's roaming agreement with new French 3G operator Free Mobile, whose cheap all-you-can-eat voice and data plans have often been accused of overloading the Orange network.
"There's no capacity problem on the Orange network," Richard said.
Philippe Chicaud, responsible for deployment and maintenance of fixed and mobile networks at Orange, said the affected system was the New Generation Home Location Register. The NG-HLR was supplied by Alcatel-Lucent, he told Réseaux et Télécoms, a partner of IDG News Service
The register, responsible for locating subscribers, consists of three elements: the front end, the central servers, and the databases. "There were some inconsistencies in the data, and erroneous messages were exchanged between the different elements. That caused some saturations, leading to a snowball effect that blocked the system," he said.
Orange customers who kept their phones turned on and didn't move between cells remained unaffected for longest, as no changes were made to their entries in the HLR. But even those customers were affected in the end, as a build-up of error messages caused the HLR to saturate.
Service resumed in the early hours of Saturday morning, with voice and SMS service restored at 12.30 am, and data by about 3.20 am.
The nine front-end servers distributed around the country, and six back-end servers, were cleaned up in turn. "We've learned how to deal with it now, and if it happens again we have procedures that will help us deal with it more quickly," said Chicaud.
Richard said he had ordered a full audit of the company's fixed and mobile network security procedures, but added that there could never be zero risk.
The last time France Télécom experienced a similar outage was in 2004. Then, the problem affected fixed and mobile networks, but the impact on customers was less, said Delphine Ernotte Cunci, executive director of Orange France. "But it's the importance of mobile networks today that led the minister for economic recovery, Arnaud Montebourg, and the minister for the digital economy, Fleur Pellerin, to stay with us until midnight and give their support to the team at the network operations centre," she said. "That, and the fact that the state is still a major shareholder in France Télécom," she added.
Other operators have experienced outages of a similar magnitude, said Chicaud, citing T-Mobile in the US.two years ago, and Verizon, O2 and AT&T. "It's not always the HLR that's at fault," he said.