Good measures can greatly improve an organisation's ability to identify problems and opportunities and obtain tangible results from its improvement efforts.
The right measures can provide feedback to change behavior and the necessary guidance for determining priorities and optimising resource allocation.
IT managers and business analysts might be interested in answers to the following questions (among others):
- How effective are our business analysis processes?
- Are our business analysts documenting requirements in a format that promotes clear, complete, and comprehensive understanding of what needs to be built?
- Are we better off in requirements management than we were this time last year?
- Which groups need more business analysts? Which ones have a surplus?
Clearly, questions like these cannot be answered with precise, unequivocal numbers. On the other hand, measuring intangible concepts is far more common than most people appreciate :
Even highly subjective factors like the quality of ice-skating and movies are routinely measured by knowledgeable judges in these fields. While someone may find fault with any particular instance, these measures are reliable guides once the user understands what they represent.
Following this reasoning, it's not difficult to think of indicators derived from countable items that could be used to obtain answers to questions like the ones listed.
For question #2, for example, data regarding the quality of the requirements produced by an analyst can be collected via inspections of requirements specifications, with requirements defects being counted and classified into various categories: missing, erroneous, unnecessary, incomplete, ambiguous, etc.
Other types of measurement include the number of change requests due to incorrect requirements (suggested by Adam Feldman), and the cost to fix post-release problems caused by faulty requirements.
In spite of the potential benefits from performance measurement, studies show that more than half of measurement programs do not continue beyond their second year, and that two in three metrics implementation fail .
Many teams struggle with how to implement an appropriate measurement program capable of providing the information needed to manage their projects and organisations more effectively, and find it difficult to avoid the problem of dysfunctional behavior (e.g., a knowledge worker inappropriately optimising one work dimension at the expense of the others, or reporting fabricated data, in order to tell managers what they want to hear).
Should a standard set of performance measures be developed and shared across organisations?
The answer is no. A compilation of performance measures from different companies would certainly be useful to illustrate how these companies measure quality variables related to the business analysis work, and to stimulate thought about what should be measured.
Prescribed solutions and mirroring other company's programs, however, are not good approaches, as an external entity does not necessarily understand or share another company's concerns and goals.
Here's an example: for a software development provider that has to estimate the price of a project based on the size of the effort, having functional requirements decomposed at a consistent level of granularity may be extremely important, since knowing how many "requirement units" are going into a release facilitates the estimation of the effort involved in developing and testing the solution.
Measuring the conformance of functional requirements with a standard level of granularity would be useful in this case, and completely irrelevant in a different organisation following a process involving less documentation and more face-to-face communication of requirements.
A measurement system is a mathematical model of something (a company, a business unit, a functional area, an individual, etc.). For this model to be meaningful and actionable, it must reflect reality (i.e., it should emulate results people see happening by other means).
As performance improves, inputs, processes or outputs change, and quality problems are solved (no longer requiring some types of measurement), changes must be applied in what gets measured, how variables are measured, how often measurement should occur, and how the information is reported.
Each organisation interested in measuring the performance of their business analysts (among other knowledge workers) should start to grow its measurement culture by selecting a fairly small, balanced set of performance measures. These measures should make sense to the business and the people being measured, and be capable of measuring several complementary aspects of the BA work, such as quality, effort, size, and schedule.
 Operational Performance Management: Increasing Total Productivity. Will Kaydos, CRC Press,1999.
 Implementing a software measurement program in small and medium enterprises: a suitable framework M. DÃƒ¬as-Ley et. al. IET Software, 2008
Adriana Beal received her B.S. in electronic engineering and an MBA in strategic management of information systems from two of the most prestigious graduate schools in Brazil. She splits her time working as a Lead Business Technology Consultant for ThinkBRQ (a NY-based consulting firm servicing clients in the financial and telecommunication industries) and 2wtx (a small agency offering web presence strategy services and business analysis resources).
By Alex Papworth
Alex Papworth is a freelance business analyst and a director of the UK chapter of the International Institute of Business Analysis.