Several Virginia state agencies continue to experience problems with data access due to an outage related to problems in a storage area network (SAN) that began last week in a data centre run by outsourcer Northrop Grumman.
An automated phone message from the Virginia Information Technologies Agency (VITA) says that as many as seven key bureaus, including the Department of Motor Vehicles and the Department of Social Services, are having problems accessing applications, shared folders and other data stored on servers in the commonwealth's Enterprise Solutions Center in Richmond.
VITA has offered updates on its attempts to repair the outage on its website. VITA's website stated that repairs to the storage system's hardware are complete, and all but three or possibly four agencies out of the 26 agency systems have been restored. The agencies were performing verification testing on Monday.
According to published reports, computer systems came to a halt last Wednesday due to a memory card failure in a SAN. A backup SAN that was supposed to act as a failover system then also experienced problems.
Earlier this year, after a long history of problems within VITA, Virginia's governor placed its CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who had co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems. Neither VITA nor Northrop Grumman had responded to a request for comment at deadline.
VITA's CIO, Lem Stewart, was fired last year. According to published reports, Stewart had withheld $15 million in payments to Northrop Grumman due to his belief that the service provider failed to meet contractual obligations. Through a $2.4 billion contract, Northrop Grumman runs the state's data centres, help desks and other IT operations. The contract requires Northrop Grumman to keep VITA's data centres in the state.
VITA's contract with Northrop Grumman has come under heavy criticism in the past for a number of problems, including project delays, cost overruns and performance issues that included other service outages.
After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrup Grumman was modified, resulting in more stringent performance requirements and greater conditions around accountability. The contract, however, also increased payments to Northrop Grumman by $105 million over nine years.
Along with VITA, the latest outage affected the State Board of Elections, the Department of Juvenile Justice, the Department of Motor Vehicles, the Department of Social Services, the Department of Taxation and the Department of Environmental Quality.