It takes a lot to shock Chris Goggans; he's been a pen (penetration) tester since 1991, getting paid to break into a wide variety of networks.

But he says nothing was as egregious as security lapses in both infrastructure design and patch management at a civilian government agency - holes that let him hack his way through to a major FBI crime database within a mere six hours.

Goggans, currently senior security consultant at security firm PatchAdvisor, says his adventure started when, during a routine network scan, he discovered a series of unpatched vulnerabilities in the civilian government agency's Web server, as well as other parts of the enterprise.

Goggans used a hole in the Web server to pull down usernames and passwords that were reused on a host of enterprise systems. In those systems, he found further account details that allowed him to get Windows domain administrator privileges -- a classic escalation-of-privileges attack.

Using this privileged access, he was able to gain full control of almost all Windows-based systems in the enterprise, including workstations used by the on-site police force. He noticed that several police workstations had a second networking card installed that used the SNA protocol to directly talk to an IBM mainframe.

By covertly installing remote control software on those workstations, he found programs on their desktops that automatically connected the workstations to the FBI's NCIC database. "That software, coupled with a keystroke capture program, would allow an attacker to grab the credentials needed to log into the FBI's National Crime Information Center database," he says.

Like most vulnerabilities he's found over his years of paid ethical hacking, this one could have easily been eliminated with some basic security strategies, he says. For instance, the police network should have been firewalled off from the main enterprise network, and the investigators' workstations kept out of the larger domain.

Also, he says the agency should not have allowed those workstations both NCIC and general enterprise network access, since they were connected to something with such obvious national security implications. Finally, the system administrators should have monitored and blocked the common reuse of passwords.

Chris Nickerson, CEO of Lares Consulting, is also amazed by the simplicity of most hacks -- especially in this era of compliance, which should demand tighter controls. In fact, he says when he was sent to do testing for a Big Four company, he was able to immediately gain full administration access to all the organisation's applications.

"This was a company that had maintained they were Sarbanes-Oxley compliant for several years. Yet I had control of the business within the first 20 minutes. I could actively change general ledgers and do other critical tasks," he says.

He also has found problems with companies that claim to be in compliance with the newer Payment Card Industry (PCI) standard. "I've had people who have spent millions of dollars on security to say they are compliant, and I walk in and pop open their main credit card processing system within 10 minutes."

The problem, he says, lies with compliance rules themselves. "The government has narrowed the scope of compliance so much to make it cost affordable that it overlooks a lot of things that are real-life security vs. paper security," he says.

He encourages his clients to focus on two technology tasks: managing patches and hardening their operating systems. "You should always make sure you're up to date on patches and turn off ports and services you're not using."

Nickerson is also a fan of automated penetration-testing tools , such as Core Security's Core Impact. "I like to show people, through the use of software like Core Impact, how easily I can get through their whole network. I even let them drive the tool so they can see how someone with zero knowledge can attack them. That's usually when they realise security is something they have to do," he says.

He recommends that even after the initial testing is done, organisations continue to use the automated penetration tools to audit their environments to pick up problems with new applications or configuration changes.

But as good as automated tools are, they are not a panacea, Nickerson says. He uses a combination of penetration testing and manual hacking to show users how easy it is to blend social engineering and network vulnerabilities to gain access to high-stakes data.

For instance, he says he once showed a luxury-car dealer that by mixing information gleaned from a phishing scam targeted at his clients and holes in the back-end architecture, he could potentially pull cash from the customers' bank accounts. "That's worse than if I told him I could steal $100 million worth of cars because that's his reputation," he says.

Brad Johnson, vice president of security consultancy SystemExperts Corp. in Sudbury, Mass., agrees that IT teams should expect penetration-testing teams to use both automated and manual tools to assess their environment. "With the automated tools, you could tell if someone ran a scanner against your firewall. But Web servers pose a bigger challenge," he says.

He adds that Web security is one of the biggest problems facing IT teams because developers are under tremendous pressure to get code out the door. "Often, developers are more concerned about functionality than the people who would abuse that functionality," he says.

He points out that there are myriad places where data being processed through a Web application can be corrupted, including at the browser level, between the client and server, at the front-end server, back-end server and during storage. Rarely do organisations test the security of data at each of these points before the application is deployed, Johnson says.

For instance, Johnson did a penetration test on a small banking institution and found that they included user IDs as part of the bank account URL. A hacker only needed to change a bit of the URL to access another bank account.

Unfortunately, this situation is not unique, he says. "Well over 50% of the Web applications in production that we test can perform cross-account actions such as logging in as user A and looking at data reserved for user B, or executing functions that only user B is authorised to do. This is just bad access control enforcement," he says.

However, he says that IT should avoid pointing fingers at Web coders and instead insert themselves into the process to ensure that applications are deployed safely.

It took some very public scandals, including a takedown of the government's Web site and published descriptions of vulnerabilities in the voter registration site, for the Commonwealth of Pennsylvania's IT team to be able to free up the budget for penetration-testing tools and beef up security for its Web development practices.

"In government, there's a big push for e-government and that's great because we should be giving citizens access to resources. But there's not enough testing of these new Web applications before they are deployed, and yet they have a huge door called Port 80 that's not secure," says Robert Maley, the commonwealth's chief information security officer.

Maley, who came onboard almost three years ago, says he had been pushing for increased penetration testing of all systems but was told the technology and human resources required were too expensive. He was able to squeak a few dollars out of the budget to buy an automated tool and train his team to run it against the government's 80,000 endpoints and 100,000 business partner connections.

But earlier this year, five portal Web sites were breached with a SQL injection launched from China. The government's main Web site was down for six hours, making local and national headlines. Maley used his penetration-testing tool to do a post-mortem on the attack and shore up any other holes. Then, a month ago, the commonwealth came under fire again when someone published a vulnerability in the voter registration database that allowed citizen data to be viewed.

"That bad press was the final thing I needed to eliminate any pushback and to create a sea change in the culture here," he says. Although there is still not enough money to bring in outside consultants, Maley is working closely with his own security team to test application code in development and in production and to train developers on security practices. "We have checks and balances on everything we do now," he says; "for instance, before a site goes live, we do penetration testing against the hardware, software, operating system and application itself."

Ready to get started? We've got five steps to successful and cost-effective penetration testing -- and five free pen-testing tools to check into.