Network admins with too much control are a common problem - and one that needs fixing, CIOs warn.
The City of San Francisco's IT department has been in the news recently, but it is certainly not unusual when it comes to allowing just one person to have unfettered rights to make password and configuration changes to networks and enterprise systems.
In fact, it's a situation fairly common in many organisations - especially smaller to medium-sized ones, IT managers and others cautioned in the wake of the San Francisco incident. Terry Childs, an employee working for the city's IT department, used his privileged access to lock everyone out of a crucial network for days.
A network administrator working for San Francisco's IT Department of Telecommunications and Information Services (DTIS), Childs was arrested on 13 July for allegedly tampering with the city's FiberWAN network. He is also alleged to have planted network devices that enabled illegal remote access to the FiberWAN network, which carries almost 60 percent of the city government's traffic.
He was jailed on a $5 million bond after refusing to divulge the passwords he had used to block access to the network. Childs pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week and asked for his bail amount to be lowered.
At a bail hearing on Wednesday, a San Francisco Superior Court Judge refused to lower the bail, even though Childs in a dramatic move earlier this week disclosed the passwords to Mayor Gavin Newsom in a jailhouse meeting. His next hearing is scheduled for September.
The episode and the city's struggle to regain full access to its locked network have highlighted the dangers involved in handing over too much administrative control of networks and IT systems to a single individual.
But the situation exists more often than imagined, said Matt Kesner, chief technology officer at Fenwick and West, a San Francisco-based law firm. "It's probably more common than we'd like to think. This is the kind of nightmare I could see happening at many organisations," Kesner said, citing his own experience from working at other companies.
"There often is a single networking guru who really does have the keys to everything. You have to work very hard to make sure that more people have the keys," and that there's infrastructure and processes in place to enforce it, he said. Often though, companies simply don't have the resources or the skills needed to really do this.
"Unfortunately it is not that uncommon to come into a situation where one or two people have created a situation where not only are they the only ones that know what is going on, but they are the only ones that can do anything," said Lou Michael, director of network and infrastructure services in Virginia's Arlington County department of technology services.
The issue isn't just control over passwords, but also over documentation relating to configurations and changes. Often in situations such as this, "requests for access, passwords and documentation are frequently taken as hostile acts by those that have been holding the keys to the kingdom," he added. "In my experience I have encountered this type of situation on more then one occasion," he said.
In one incident, a mainframe systems programmer had to be fired for changing access rights because he disapproved of others' activities on the system, Michael said. In another case, the individual resigned when he "realised that the pressure to follow processes and procedures was not going to go away despite the protesting," Michael said.
These practices persist due to lack of resources and prioritisation, said Richard Gorman, CEO of Vormetric, a vendor of database security and encryption products. "For many organisations, security is not a mission-critical priority until it has been breached," Gorman said. As a result, it is not unusual to find many companies handing over control of entire networks and systems to one individual. "There is no valid technical reason to do this," and it is something that can always be avoided. Nonetheless, it is "surprisingly common."
Especially in smaller and medium-sized companies, control is vested in a single individual in order to more cost-efficiently troubleshoot problems and take care of daily administrative tasks such as resetting passwords, said Raj Rajamani, product manager at Solidcore Systems, a vendor of change management products.
"If you have one person serve as an administrator, then have another person audit the administrator, and have yet another person audit the auditor, you get into a costly and time-consuming cycle of inefficiency," he said. Tools are available to do this sort of auditing, but often the process can be more of an impediment than a benefit, he said.
"Single points of failure are always bad," said John Pescatore, a Gartner analyst. "There should never be one person who is the only person who knows the configuration or the password." Companies need to make sure there are at least two if not three people who share the knowledge of network configurations and server configurations.
"As a minimum, require it to be documented and stored somewhere if personnel limitations say you can't have personnel with overlap," Pescatore said.