Water pumps need firewalls too. That's what operators of the Tennessee Valley Authority's (TVA) Browns Ferry Nuclear Plant discovered last August when they were forced to manually shut down one of their plant's two reactors after networking problems caused two water pumps to fail and threatened the stability of the plant itself.
On 19 August 2006, operators found themselves in a potentially dangerous “high power, low flow condition", when one of the plant's two operating reactors was not recirculating enough water to properly cool itself, according to a report by the US Nuclear Regulatory Commission (NRC). Operators were forced to perform a shutdown of the plant.
Built in 1974 in northern Alabama, Browns Ferry was once the world's largest nuclear reactor.
Although the Browns Ferry incident was not anywhere close to a nuclear meltdown, it was a serious situation, said Eric Byres, chief executive of Byres Security in the US. "They realised that their recirculation system wasn't working and they were in danger of something undesirable happening," said Byres, an expert in industrial systems security who was consulted on the matter.
The cause of the pump's failures? "Excessive traffic" on the closed Ethernet network used by the plant's control systems, the NRC said.
The NRC report said the origin of this excessive traffic was unclear, but Byres suspects that the problem was due to faulty networking code the controllers used by the plant's recirculation pumps. They may have suffered from the same well-documented networking flaw that has taken down similar systems in food processing, steel and pulp plants in the past, Byres said. "I'm personally aware of at least a dozen incidents at this point that relate to this particular fault," he said.
Although he declined to name the manufacturer of this product, Byres said that it has a known bug that can cause a crash by generating too much networking traffic. "It's like the loud guy at the bar standing at the table," he said. "It kind of cuts down on the ability of everyone else to have a decent conversation."
After the incident, Browns Ferry's operators began developing firewalls for the different controllers on their network as well as a network firewall device to limit the traffic between devices within the plant's internal network, the NRC said.
Two members of the US House of Representatives wrote to the chairman of the NRC expressing their concern that the Browns Ferry failure may have been due to an outside attack. "Without a thorough, independent review of the logs and associated data, the assumption that this incident is not an outside attack is unjustifiable," they wrote.
Byres agreed that a more thorough investigation would be useful. "It's unfortunate that they didn't dig in further," he said. "These types of glitches are well known."
Virtually no control systems use firewalls at present, although that is starting to change, Byres said. "The idea is still in its infancy, but - particularly in the oil and gas industry - there's starting to be a lot of interest in doing this sort of defence-in-depth," he said.
"The secretary at that nuclear plant has a firewall on her desktop, but we'll put a [control system] out there that's absolutely mission-critical and that has absolutely no protection on it."
Find your next job with computerworld UK jobs