I’ve gotten various comments in response to the software productivity/quality metrics that I cited from Michael Mah’s presentation to the New York SPIN chapter the other night — ranging from “Of course! It just proves what I knew all along” to “Baloney! I don’t believe a word of it!” All of this reinforces one of the primary themes that Michael emphasized in his presentation: the statistics should motivate us to do some thinking and investigation into our own projects, and our own development approaches.

Here are three comments that I’ve gotten from several people:

1. One offshore outsourcing project, and two “agile projects are just isolated data points, not necessarily a trend. True, we need lots more data — and we’ll probably see larger studies in the coming months and years from metrics researchers like Michael Mah, Capers Jones, and Howard Rubin. But in the meantime, every project manager and every IT organization has to carry out its own process of “generalization,” by asking whether these three case case studies are similar to, or dramatically different from, their own projects. The question is not “what absolute facts do these case studies illustrate?” but rather “what insights and perspectives do the case studies offer us?”
2. Rough, self-reported statistics are potentially inaccurate, and perhaps completely incorrect. Well, maybe … but I would tend to trust veteran metrics experts like Mah, Jones, and Rubin to spot the cases of self-reported productivity/quality figures that are completely bogus. Still, it’s true that if you ask a project manager questions like, “How many people did you have on your project during the past year, and how many person-months of effort were required to deliver a working system?”, you’re only going to get approximations. The project manager may have forgotten that someone joined the project for two months and then disappeared; and he/she may have overlooked (or consciously ignored) all of the unpaid overtime hours worked by the team. Even questions like “How many lines of code did you deliver, in the final working system?” may be answered with approximations like, “Oh, I think it was about 200,000 lines of code,” rather than a precise figure that should be easily obtainable from an automated tool. Consequently, the productivity (lines of code per person-month) and quality (defects per line of code) have be interpreted as approximations; you can’t expect accuracy to six decimal places.
3. Why were all of these metrics gathered at the end of the project? Why not capture interim figures throughout the project, and estimated figures at the beginning of the project? Good question! Well, postmortem figures are better than nothing, because they can be used to help explain the actual success or failure (e.g., did the project take longer than expected? did it have more, or less, defects than expected?) that was actually observed. To the extent that the metrics are perceived as credible — and more important, to the extent that the metrics-gathering process is seen as credible and beneficial — it should be possible to persuade project managers and higher levels of IT management to invest the necessary time and effort to collect metrics throughout the project (in order to assist whatever mid-course corrections are needed), and estimated figures at the beginning of the project.