For several years now, SOX has pushed IT to work with financial and operational auditors to ensure that technical implementations of security and privacy controls actually do what's intended. With this assurance in hand, officers of public companies can sign attestations to SOX compliance without nightmares of Enron-style federal vacation packages.

Most public companies have found their way to SOX compliance in one way or another, with the result that we're unlikely to see another massive corrupt meltdown of a public company in the near future. But the ends do not provide a solid defence for confused and incomprehensible means, nor a justification for auditors and consultants drawing out the process for their own gain.

If one looks back a few years, Boeing seemed to understand the compliance process, and had it sufficiently under control to talk about process improvement in the same breath. Boeing's Controller, Harry McGee said: "Beyond the strict regulatory requirements, by reviewing and testing our internal financial controls we are planting seeds for continuous process improvement and helping change the way Boeing does business in the future."

Why then is Boeing now being held up in the press for ridicule? Had they gone the route of most companies - choosing a single audit firm to work with financial and IT control owners - they might have gotten consistent audit results. Wildly subjective results, if experience serves as any reasonable sample, but consistent from year to year and solid-looking.

But Boeing engaged several audit teams and consultants, and the inconsistent results warranted more investigation than most organisations subject themselves to. By being more diligent, Boeing discovered things about itself that most public organisations gloss over.

Under scrutiny

The intent behind SOX is to prevent or expose corrupt practices, with the sections applying to information systems commonly interpreted as flexible enough to deal with the pace of technology.

Dennis Brewer from says: "The Sarbanes-Oxley Act's call for 'adequate internal controls over financial reporting' is vague, and for good reason. By withholding prescriptive details, the regulators created a moving target that allows compliance requirements to increase with advances in technology."

Nice theory, but it doesn't work out in practice. From the standpoint of preventing corrupt practices, SOX is a moderate success. However, what if one examines consulting dollars and the functional disarray that many SOX audits leave in their wake? When one year's financial and IT control audit process begins to blend into the next (and the next) - requiring a level of record-keeping and analysis effort that exceeds core business practices - the reality is sobering.

In that light, SOX begins to look more like a quagmire for the incompetent, and a contrivance for highly lucrative audit and consulting business. In retrospect, McGee's seeds of continuous process improvement look hopelessly optimistic and naive.

Variances in the variances

I'm not knocking the intent of Sarbanes-Oxley, only the idea that vague directives are desirable. There's a lot to be said for having a sober assessment of whether technical controls - IT systems and network in most companies - accurately transform financial management requirements through security and privacy policy into an implemented reality. By identifying assets and controls, then testing for and finding material weaknesses, it's possible to get a sense of whether the controls work, and documenting what's to be done when they don't.

But it's only a sense. Despite the collaborative efforts of industry and quasi-official groups such as the Public Company Accounting Oversight Board (PCAOB) to standardise compliance assessment and audit metrics, there are significant differences in both which and how measurements are made.

PCAOB suggests using the Committee of Sponsoring Organisations of the Treadway Commission (COSO) framework, and many audit forms prefer to adapt Control Objectives of Information and Related Technology (COBIT) from the IT Governance Institute. Different standards such as SAS 70 Type II from the American Institute of Certified Public Accountants (AICPA) or WebTrust criteria are used to assess outsourced services. Still other auditors and assessors are just making it up as they go along because they feel special, or just don't know any better.

Any auditor, assessor or consultant who presents a SOX report with compliance levels in the form of a percentage with significant digits to the right side of the decimal point is ether deluded or lying. With significant variances in the value of assets, nature and state of technical controls, and speculative impact of weaknesses, one might as well predict daily rainfall to a third or fourth decimal. The variance in those values even from one side of a building to the other renders such statements suspect if not truly meaningless. Those who utter them ought to be politely shown the door.

In moderation

From the standpoint of regulators, auditors and consultants, SOX looks like bleach: The more you pour in, the cleaner and clearer things get. But responsible executives need to look at the fabric of their business and determine the appropriate level of SOX assessment: somewhere more than made-up metrics, but stopping short of making the business unravel.

Boeing's response reflects this, and ignoring auditors quibbling over a compliance point or two is the right thing to do. Specifically, they say "Being [SOX] compliant does not require zero deficiencies. It requires that mechanisms are in place to effectively identify, evaluate and remediate any deficiencies to the internal control structure. It also requires that appropriate reporting occur based on the significance of those deficiencies. And it requires that deficiencies are remediated within a reasonable time period." They go on to politely indicate that the variance in compliance metrics between audit firms is not a significant finding.

It's OK to push back on assessors and auditors - and not only the ones with dubious methods - to ensure that the depth of analysis reflects the right level of risk from compromised processes or failed technical controls. This is important as financial and process control auditing begins to spread from business partners to service providers around the globe.

If basic business processes are undocumented, deeper assessment and discussions about repeatability and roles are warranted. However, if last year's results indicate investor's funds appear to be consistently where they ought to be and executives aren't lining their pockets with employees' 401k plans, there's little need for the SOX auditors to be talking about function point analysis over lunch with developers.

Jon Espenschied has been at play in the security industry for enough years to become enthusiastic, blasé, cynical, jaded, content and enthusiastic again. He is based in Seattle, where his advice continues to be ignored by CEOs, auditors and sysadmins alike.