How Treasury and AT&T botched billion-dollar network upgrade

Three years ago, the US Treasury Department awarded a billion-dollar network consolidation project to AT&T, which claimed to win the deal due to its superior "transition plan."


Three years ago, the US Treasury Department awarded a billion-dollar network consolidation project to AT&T, which claimed to win the deal due to its superior "transition plan".

Today, both the federal agency and its carrier admit the Treasury Network (TNet) - which involves consolidating 15 separate networks into a single, secure IP backbone - is notably behind schedule and continues to experience performance problems.

Indeed, TNet has encountered so many issues that the umbrella program that funds it - which goes by the acronym ITT TSS for IT Infrastructure Telecommunications - recently landed on the Office of Management and Budget's list of 26 high-risk federal IT projects. Federal CIO Vivek Kundra released the high-risk list a month ago, threatening to discontinue any of the projects that can't be turned around.

How did TNet get so far off track?

In September 2007, TNet was the first major award under Networx Universal, which is a governmentwide telecommunications program managed by the General Services Administration. AT&T won TNet in a competitive bid believed to include Networx Universal rivals Verizon and Qwest.

TNet is approximately two years behind its original schedule. The transition of the Treasury's predecessor data network -- called TCS for Treasury Communications System -- was supposed to begin in November 2007 and take about a year, according to a January 2010 report by the agency's Office of Inspector General. Instead, the agency just finished migrating the TCS circuits to AT&T's Networx contract in August 2010.

Treasury blames the delays on new network security requirements that were put in place by the Bush Administration in November 2007, shortly after TNet was awarded.

These requirements - dubbed the Trusted Internet Connection (TIC) initiative -- mandate that all federal agencies deploy a standard set of security tools including antivirus, firewall, intrusion detection and traffic monitoring on their networks to protect against hacking attacks launched by foreign governments and criminals.

"Treasury's TNet transition was complicated by the introduction of additional requirements introduced post-award (e.g., the Trusted Internet Connection initiative)," said Diane Litman, Acting Deputy Assistant Secretary for Information Systems and Chief Information Officer of Treasury, in a written response to questions. "Treasury was also challenged in melding different operational models of the legacy telecommunications systems of our bureaus into a single model compatible with the managed service nature of TNet."

Treasury said it has made progress in getting TNet back on schedule, but that it still hasn't reached its goal of consolidating all of its networks onto a single WAN.

"A concerted effort on the part of Treasury, its bureaus, and the vendor (AT&T) culminated in the migration of its legacy data telecommunications circuits to Networx in August 2010," Litman said, adding that the legacy data circuits were disconnected by the end of September.

"This marks a major turning point for the investment and will enable Treasury to turn its attention to implementing the Department's IT strategic direction of voice, data and video convergence, service and data center consolidation, and cloud computing technologies to achieve cost efficiency," Littman said.

A complex undertaking

Both Treasury and AT&T agree that some of the TNet delays stem from the complexity of the project.AT&T was tasked with building a secure IP Multiprotocol Label Switching (MPLS) network spanning 800-plus locations and supporting more than 100,000 employees of the Treasury Department's 12 bureaus, which include the Internal Revenue Service, the U.S. Mint and the Bureau of Engraving and Printing.

TNet will provide these bureaus with a single, converged backbone network to carry data, voice and video traffic. It also will feature secure remote access to support teleworking as well as disaster-recovery services.

The initial TNet award was for $270 million, but AT&T officials said the deal could be worth as much as $1 billion, including add-on services bought over the 10-year life of the contract.

When it won TNet, AT&T called the award a "vote of confidence" for its Networx Universal offerings and vowed to meet Treasury's needs with "a host of services."

Today, AT&T has little to say about the status of TNet and the fact that it is on Kundra's high-risk list.

"During the past three years, AT&T Government Solutions has worked with the Department of Treasury to build and transform the Department's network into an advanced, next-generation enterprise network," AT&T said in a brief statement. "This new network is a complex undertaking, as it integrates 15 disparate networks into a single, consolidated secure enterprise network."

Although Treasury and AT&T have made progress on TNet, the troubled network consolidation effort is not out of the woods.On the Federal CIO's IT dashboard, Treasury admits as of Sept. 1 that the transition to TNet "experienced notable schedule delays, which resulted in pent-up demand for additional services. TNet completed the transition from the predecessor system, TCS, in August of 2010. However, bureaus are experiencing post-transition performance problems."

Also on the IT dashboard, Treasury said it was preparing an improvement plan for TNet that will be reviewed by OMB. "The Improvement Plan will address issues regarding governance, security, configuration management, managing the backlog of new requirements against long-range requirements for operations and maintenance, and the key role of TNet in the Federal Data Center Consolidation initiative."

Next up for Treasury is migrating its voice circuits off of the expiring FTS 2001 contract to AT&T's Networx contract. Treasury says it will reach that milestone by the end of September 2011.

Treasury's checkered past

TNet isn't the first time that Treasury has run into trouble with one of its network procurements.

In December 2004, AT&T was awarded a $1 billion telecom deal called the Treasury Communications Enterprise (TCE) contract. The TCE award was later overturned after five of the six other bidders protested the award to AT&T.

Treasury decided to wait and purchase its new data network through the federal Networx program, which was being procured at the time. The agency put out a task order for TNet under Networx Universal in the summer of 2007 and awarded the deal to AT&T a few months later.

Given Treasury's track record with network procurements, one federal IT industry observer who requested anonymity guessed that Treasury's IT staff is at fault.

"My intuition says that Treasury doesn't have enough sufficiently competent program managers to execute on these complex requirements," the source said. "It's probably a combination of incompetency and inexperience."

"Recommended For You"

HM Treasury awards core IT services contract Barack Obama: US government technology is 'horrible'