Not long ago, the legal department at a financial services company in New York got a phone call from a hospital in London.

The query: Why are you hacking us? With two known IP addresses, it wasn't difficult for the financial firm's information security staff to go back through the logs looking for traffic between the two organizations.

And with the traffic identified, locating the computer from which the hacks were taking place didn't take long, either. The culprit: an individual who-as their human resources records soon confirmed-had formerly worked at that very hospital.

Ah, the good old days. As investigations go, says Winn Schwartau, founder of security awareness certification company SCIPP International and an information security expert who has testified before the US Congress, the hospital hack was an increasingly rare example of a fast-dying breed: a pure infosec forensic investigation, carried out digitally.

Of course, apprehending the suspect in such a case, or seizing physical evidence, requires a whole new dimension. And that's why CSOs and CISOs increasingly report that purely "computer" investigations, like the hospital hack, are a thing of the past-as are purely "physical" investigations.

Pretty much every significant investigation these days now includes elements of both, whether the case at hand requires face-to-face interviews, forensic accounting, e-mail discovery and review, computer and network forensics, cell phone records, video surveillance analytics, access-card logs, inventory audits or all that and more.

So in such an environment, how can CSOs and CISOs staff, train and prepare for such "blended" forensic investigations to be effective? What are the areas to concentrate on, and where do the pitfalls lie? And how, in short, can security navigate this blended investigative world?

Blended Investigations: Together Is Better

"No matter how good the forensic investigation is at the IT level, there's always going to be a physical investigation-targeted interviews, building-access logs and so on," says Robert Huff, a former FBI agent and now managing director and global leader for corporate investigative services at Aon Consulting . "Almost always, computer forensics needs to be supplemented by physical inquiries."

Likewise from the other side of the fence, adds Chris Boyd, head of forensic operations at the UK-based Detica Forensics, and a former police specialist. "The physical world is going digital," he asserts. "Access logs aren't sheets of paper, but digital records and even CCTV footage are moving from VHS video tapes to hard drives.

It used to be that IT forensics supported the physical investigation-and although there's still a place for both types of investigation, it's now the physical investigation supporting the IT one in many cases."

And in this dual "blended" world, says William Pelgrin, director of the New York state office of cyber security and critical infrastructure coordination, one thing is clear: The era of the blended investigation is not without its advantages.

For in reality, he points out, infosec investigators have long had to bear in mind that there might be a physical dimension to the investigation at hand-and likewise physical investigators.

"Trying to look at things one-dimensionally tended to introduce artificial constraints," he argues. "It was always a smart move to ask if there was a physical component to a cyber attack, and vice versa. Yes, there are pure cyber incidents, and there are purely physical incidents-but it's wrong to assume that's what they are without exploring the possibility that they might not be. You have to look at things from different angles to get the complete picture."

And the importance of this recognition, he stresses, isn't just that more bad guys get caught. Instead, it's that with the need to be multidimensional out in the open, investigations can appropriately "tool up" from the start.

"In today's world of investigations, you can't do-or be-everything, so you bring in the skills and competencies that you need, as and when you need them," explains Pelgrin.

But which precise skills and competencies? During the first few minutes of an investigation is where it's most critical to get things right, and it's here that appropriate training is often required, says David Brown, managing consultant for security advisory services at consultants Forsythe Solutions Group.

"The first few minutes of the initial reaction tend to set the stage for the rest of the investigation, and it's during those first few minutes that it's vital that the physical guys understand the requirements of the IT team, and vice versa," he emphasizes. "There's a balance to be drawn between incident mitigation and preservation of evidence-and that balance often depends on the organization in question-but each team needs to know which actions will help the other team, and which will hinder them."

On a related point, understanding each other's preferred modus operandi is also useful, adds Adrian Davis, a London-based senior research consultant at the Information Security Forum , a not-for-profit international association of some 300 leading international organizations.

"Physical security people tend to approach investigations in a particular way, and that might seem strange to IT people," warns Davis. "It's important they understand each other's approaches, so that they reinforce, rather than conflict [with], the other party's investigative work."

Beyond that, it's also sensible for each team to understand the other's strengths and weaknesses-and how those characteristics dovetail with their own team traits. "Theft, for example, is something that the physical guys usually have more experience with-but if someone is using a computer system to divert shipments, then you'll need the involvement of both parties-and the physical guys need to know when to step back and call in the [digital] experts."

And sometimes, of course, the experts in question will be external investigators from law enforcement agencies. Rules and procedures vary with jurisdiction, but a good operating assumption is that when the investigation uncovers the fact that a crime may have been committed, local law enforcement agencies will need to be informed.

At which point, there's likely to be the need to call the human resources department, and usually the legal department as well. "The 'people' component of an investigation is always the most difficult," warns Schwartau. "People management is the remit of the legal and human resources folk, and they don't fold well into the world of geeks and geekdom."

In short, he sums up, recognize that legal and human resources people are going to show up; train investigators in what they will be looking for and the consequences of noncompliance-such as a countersuit from an employee with a grievance.]

Investigations and Law Enforcement

Just as internal investigators from the cyber and physical organizations need to understand each other's procedures and preferences, the same holds true when law enforcement agencies are called in.

While physical security people tend to be familiar with chain-of-custody requirements, IT forensics people don't always pay the attention they should to this, warns Howard Schmidt, a former CSO for Microsoft and eBay with a background in law enforcement, who these days serves on the board of (ISC)2.

In today's wired world, he points out, locking down corporate systems until the law shows up isn't usually a practical proposition. The result-internal investigators gathering evidence usually while they fix whatever the problem is-calls for schooling in evidence gathering and preservation.

"It's partly about specific training in what to do and what not to do, and it's partly about building a sense of mutual trust between the internal investigators and external law enforcement agencies," says Schmidt. "Physical investigators tend to understand this better: Now the IT people are getting trained, and they need to understand that an image of a file dump isn't as good as evidence with a full chain of custody."

New York's Pelgrin, for example, organizes annual training sessions for New York's infosec employees, where precisely such topics are covered. The issue isn't just about not impeding the investigation, or inadvertently destroying potentially valuable evidence, he stresses; it's also about promulgating clear-cut guidelines for establishing the chain of custody.

"When you take possession of a machine, or possession of a hard-drive image, which could then go on to feature on a disciplinary or court case, it's important to be able to prove in a tribunal or a court just who has had control since that possession was taken," he says. "Evidence must be presented in its original state, and with proof that tampering has not been possible."

But in the era of the blended investigation, and with physical and IT forensic investigators working more closely together, is there actually a need anymore to differentiate between the two skill sets? In short, does the era of the blended investigation bring forth the blended investigator?

The jury, it seems, is out. "Do you take people with a strong investigative background, and train them in computer forensics-or take people who have strengths in computer forensics, and try to train them in investigative skills?" asks Amit Gavish, managing director of corporate intelligence at Shelton, Conn.-headquartered security consultants SSC. "It's something we wrestle with all the time-and typically, we find that the people with the best IT forensic skills don't have the right investigative mind-set."

With some caveats, Peter Yapp agrees. Now the London-based head of network forensics at business risk consultants Control Risks, Yapp actually set up such a team when working for the United Kingdom's customs service in the 1990s.

"In establishing our computer forensic team, what we did was to take existing customs investigators and teach them IT forensics, rather than attempting to do it the other way round," he explains. "It worked, but we were probably lucky in having people with a reasonable IT background already. I'm not sure you could take any investigator and get them up to speed in forensics-just as you can't take any IT technician and turn them into an investigator."

The Investigative Mind

Indeed, Yapp argues, the "forensic" part of the job description probably obscures the essential aspect of the role that is common to both physical and IT forensic investigators: solid investigative skills.

"What I look for is someone who can speak both languages: the language of computers and the language of the real world," he says. "More importantly, though, I want people who don't give up, who look around them and observe what's going on, and who see and then act upon anomalies. It's not just about looking for keywords on a disk-it's about picking up signs that something isn't right."

SSC's Gavish concurs. The mind-set is important, he stresses-and ultimately determines whether an investigation is staffed by two specialist skill sets or one person with both. "The physical investigator will want to stay close to his comfort zone of traditional investigative approaches, while the IT forensic person is going to feel most at home with the IT tools and techniques he or she is most familiar with," he says. "If you can't break that, then it's best to double-staff and task people to do individual parts of the overall investigation."