The workhorse measures of financial return haven't really changed, nor has the maths behind them. But as fear of a recession spurs a renewed interest in value - and as the strategic mission of IT shifts from saving money on back-office functions to enhancing revenue - "return on investment" has taken on new meanings and importance.

At consumer products giant Procter & Gamble, ROI used to focus on cost savings. "ROI was about the bottom-line impact of projects," recalls Robert Scott , vice president of information and decision solutions at P&G. "Now we have moved to a phase called 'value creation,' which is a combination of bottom- and top-line ROI."

The "top-line" objective most often is sales, he says, but it could be something else, such as faster movement of cases of Tide, Pampers or Pringles through P&G's US$76-billion supply chain. The new approach, which began in 2004, was important enough to be given a name: Global Business Services -Next Generation.
As companies increasingly try to include intangibles such as customer satisfaction in their return metrics, IT and finance people are finding new ways to deal with uncertainty.

"The biggest change I have seen in five years is what I call the bifurcation of technology between the operational and the strategic," says Steve Andriole, a professor of business technology at Villanova University. "It has phenomenal implications for project prioritisation and ROI. To a great extent, Nick Carr was right - operational technology has been commoditised."

"Operational" projects - such as upgrading a network, replacing PCs or even automating a back-office function - are usually all about cost savings and are subject to fairly straightforward, tried-and-true evaluation metrics, Andriole says.

Of the metrics, and the projects themselves, he says, "IT people have gotten so much better over the past five years. The practice of the discipline has improved through use of business cases, the quantification of metrics, the use of benchmarking data and industry data."

While operational IT projects aim to save money, strategic ones try to make money, Andriole says. An example of the latter might be a project to integrate databases across business units to facilitate cross-selling. The cost of doing that could be relatively easy to estimate, but the benefits would be harder to quantify, he says.

At least the benefits of cross-selling are known qualitatively. But in some cases, the benefits from an IT investment may be dimly perceived or even unknown. For example, Andriole says, "how do you quantify the benefits of open-source vs. proprietary software, or the benefits of Web 2.0? Companies love Web 2.0, and they are deploying it like crazy. What impact is it having on collaboration or communication or business performance or customer service? No one knows."

The basics

Despite the difficulty of developing good estimates, four basic methods for comparing project costs and returns remain in vogue: return on investment , payback period , net present value (NPV) and internal rate of return (IRR). Many more have been developed, and some have briefly been in fashion, but these four metrics have endured for decades. Even so, experts warn, each of them can lure the unwary into pitfalls.

Ian Campbell, CEO of Nucleus Research Inc., is an advocate of keeping things simple. "The worst thing you could do is come up with the next trendy metric," he says. In fact, he maintains, companies really only need two measures, ROI and payback period.

But although he recommends the use of the simplest measures, Campbell does suggest tweaking them with "adjustment factors" to allow for real-world uncertainties. He advocates applying a "believability" discount factor to the estimates that go into investment return calculations -- in other words, reducing the estimated benefits to reflect their uncertainty.

For example, estimates of direct, measurable costs savings might be used at their full value. But estimates of indirect cost savings tied to increases in worker productivity might be discounted by 50%, and "very indirect savings" from increases in manager productivity would get discounted even more. That makes the resulting number more conservative and eliminates a certain amount of wishful thinking, he says.

At the same time, Campbell cautions against allowing these discount factors to be applied on an ad hoc basis. Sometimes, he says, each person in a chain of command reduces the estimated benefits in order to be conservative and not be seen later as reckless.

"The chief financial officer doesn't trust the IT department anymore because the IT department has just made up too much stuff," he explains. When discounting happens at multiple points, the resulting estimate is no longer tied to any realistic financial figures.

Instead, Campbell recommends that all the stakeholders in the process -- the analyst preparing the estimates, the project manager, the CIO, the chief financial officer -- work together to develop realistic discount factors and then apply them as consistently as possible from project to project.

Collaboration between IT and finance is just what goes on at P&G, where a person from finance is assigned to work on every major IT project. That person helps prepare estimates of cost savings, incremental sales and return on assets, the three key measures of financial success for a project at P&G.

A favourable number in any of these areas can get a project approved, but the cost and sales numbers -- cranked into an NPV calculation -- are the most frequently used for IT projects. "Any one of those creates value for the company, for our shareholders," Scott says.

At one time P&G tried to apply factors to discount the most uncertain numbers but found that the extra effort had little payoff. That's not to say that the company ignores intangible benefits in its cost-benefit models. In fact, while cost, revenues and return on assets provide an important initial filter, projects must also be seen to meet other, "softer" targets, Scott says.

Those soft targets, such as boosting customer satisfaction, increasing customer touch points or improving product quality, become part of the goals of the project and live on after the project has ended as quality measures to monitor.

The goals can be intuitive - "You know it when you see it" -- but managers do attempt to quantify them, Scott says. Sometimes a proxy measure is adopted. For example, the number of customer complaints becomes a measure of product quality.

Dealing with uncertainty

How to include these soft metrics in a financial analysis has become a matter of much debate. Some companies don't include them at all because measuring them is thought to be too hard or subject to too much manipulation, says Nick Vitalari, an executive vice president at BSG Alliance. "Companies think they can't quantify the benefits side," he says, "but that's simply not true. You have to use some creativity."

Vitalari says one way that is becoming increasingly popular is to conduct "business experiments" - which for IT could be system prototypes, possibly linked to test marketing - before proceeding with a project that may or may not deliver value. Even fairly straightforward projects, where the costs and benefits are predictable, can gain from this type of business experimentation, he says.

Experimentation happens frequently at P&G, where there is not a single go/no-go decision point for a project. "More and more, we do quick pilots, so you do a lot of learning early on before investing heavily," says Scott. "Is this really something that will create value? Does the technology work? We call it an 'innovation funnel,' where the funnel is wide at the beginning but you have gates where you weed out options as the project progresses."

The Schwan Food uses NPV, IRR and payback period to evaluate IT projects costing more than $1 million. Cost estimates are developed by IT, but the benefits estimates come from the business units, says CIO Kate McNulty. When benefits are hard to quantify, IT works with business managers to help do that, she says.

Numbers with a great deal of uncertainty are dealt with in two ways, she says. They can be the subject of "sensitivity analysis. She explains: "We might ask, 'What if my benefits come in at only 75% of what I have estimated -- what's my IRR? What if they come in at 50%?'"

The second way is to compute an "expected value," the dollar amount of an outcome multiplied by the probability of its occurring. "If I say I'll get $100,000, and my confidence in getting it is 50%, then that goes into the analysis as $50,000," she says.

Harrah's Entertainment uses three metrics to prioritise IT projects -- primarily NPV and secondarily IRR and Economic Value Added (essentially ROI less the cost of invested capital). Increased sales is usually the key benefit to be measured, but the business sponsor of a project works with IT to measure softer benefits such as increased guest visits at Harrah's hotels and casinos, customer satisfaction and even employee satisfaction.

The use of more than one metric has pros and cons, says Harrah's CIO Tim Stanley. "Using multiple criteria to assess a project provides a robust framework for decisions," he says.

"Each tool takes into consideration the investment and the expected business value or return, and we are not limited to a single point of view." But, Stanley says, the prospect of sophisticated financial analyses can inhibit some people from submitting an idea for consideration.

Indeed, the work involved in doing any kind of rigorous financial analysis can be daunting, Gardner says. "It's a blend of corporate strategy and corporate finance and high technology," he says. "Each of those fields is fairly complicated, and when you combine them, you are dealing with some of the most challenging problems a company has.

"And incentives are not always there to come to the right answer," he adds. "The incentives are to get the system built."

"In the long run ...

... we're all dead," said the late economist John Maynard Keynes to show his support for short-term government intervention in recessions. But in IT, just what is short term and what is long term? More important, how long is too long to wait for benefits from a project?

"If the return is more than three years out, don't do it, period," advises Nucleus Research's Campbell. His favourite evaluation metric is payback period, and he says three years maps nicely to the life cycle of many technologies.

Payback period is McNulty's favourite metric, too, but the Schwan Food CIO says she's willing to think out further than three years. "If the payback for a project is more than five years, I question it," she says, "and if it's more than 10 years, I won't even consider it."

But Nick Vitalari at BSG Alliance says taking a very long view is sometimes appropriate. He recalls the huge logistics system project more than 18 years ago at Wal-Mart Stores Inc. It resulted in a supply chain so efficient and innovative that, over the ensuing decade, it propelled the company to the top of the retailer heap.

"The investment seemed crazy at the time," he says. "But they took that system and used it as a platform to enable a whole new level of business performance. If they had looked at it just on the basis of ROI or payback period, they wouldn't have seen the long-term portfolio of options that they were creating."

The importance of post-mortems

McNulty says the biggest change she has seen in IT cost-benefit analyses in recent years is a new rigor in doing project post-mortems and putting lessons learned from them back into the planning process. "In the past, a lot of business cases were put together, but they weren't validated after implementation," she says.

The project post-mortems at Schwan Food are done jointly by IT and the business units. As an example of something learned from this process, McNulty says, "When you are defining a project, make sure you have distinct baseline numbers." If multiple simultaneous projects all have improving customer retention as a goal, it won't be clear at the completion of any one project just what led to a change in that customer-retention metric. "So you want to uniquely quantify the benefit and exactly tie it to the initiative," she says.

Far too few companies do such project post-mortems, says Christopher Gardner, a co-founder of consulting firm iValue. He says a notable example of a company that does them religiously is The Boeing, which has for years "been very careful about learning from mistakes."

(In fact, Boeing pioneered applying the concept of learning curves -- in which performance steadily improves as learning occurs -- to its financial and manufacturing processes in the 1930s.)

But, he cautions, if employees know that heads will roll whenever subpar performance is revealed, they will resist post-mortems on the projects that need them most.

"It takes discipline and a management open to learning," Gardner concludes.