IT Measurement - Desk Reference

The second lesson is that interesting does not equal valuable. IT often delivers metrics that are interesting, cutting-edge, or creative. Early in the program, they may appear to be working because they generate discussion among management or trigger a revelation. Management may even fix a previously unrecognized technical problem or two revealed by the measurement. But the real test is whether the metrics provide a positive return on investment, and the answer almost always is no.

The third lesson is that, within a typical measurement program, performance data alone may not be enough to generate value. Some, but not all, performance needs may be satisfied by a traditional metrics program. Questions that may need answers, which a metrics program by itself cannot provide, include: Is the enterprise improving over time? What are our strengths and weaknesses in this area? How are we performing against our commitments? What value is the technology organization providing?

The fourth lesson is that, whatever it may be and however it is designed, the measurement system will evolve over time. The more valuable it is, the more it will evolve. As problems it identifies are solved and as issues it initially focuses on are resolved, it will need to refocus on new questions that in many cases are prompted by the initial measurements. The program will need to present information in new ways that better fit the changing needs of managers as they become more sophisticated in the use of measures. Thus, the driving force behind this evolution will be the rapid learning curve organizations experience when presented with effective measurement.

The fifth lesson is that measurement costs money and other resources, and the enterprise has only a limited amount of those resources to dedicate. Measurement solution planners may not know what that limit is, but they will find out rapidly when they reach it.

Lesson six is that performance data always involves "guesstimates" -- and making it more accurate usually carries a price tag. This relates directly to the issue of moving beyond Phase 2 in the sophistication of the measurement program. Gathering rigorous data at any level usually requires a fairly large initial investment, and even then, the data quality will only improve gradually over time.