I gave a presentation on death-march projects (see “Death march projects are back“) this afternoon at the Software Best Practices conference here in Albany, and it provided the opportunity to hear several other presentations during the course of the day — including Tony Salvaggio, Robert Lawhorn, and Joe Hessmiller of Computer Aid, Inc., and Herb Krasner, founder of the Software Quality Institute, and a well-known expert in the area of software process improvement. There were some common themes in the presentations:

• the software engineering recommendations that we were all giving were essentially the same as the ones we gave throughout the 1990s (Krasner and I spoke at several conferences together during that period).
• things haven’t gotten noticeably better; software projects are still behind schedule, over budget, and they deliver systems that are full of bugs and difficult to maintain.
• the recommended guidelines and strategies for developing good software are not rocket science, and they don’t require a Ph.D. in computer science or software engineering. For the most part, they’re common sense, or a compilation of “Management 101″ guidelines.
• none of us can figure out what it will take to get IT organizations to stop procrastinating and/or ignoring the advice, and actually doing something to improve their software processes and deliver better software.

I should withhold judgment at this point, because I haven’t seen any recent figures to confirm the popular suspicion that software productivity and quality are still as bad as they were a decade ago; I’ll be speaking at another venue of this same conference on Thursday (Sep 14th), where I’m likely to hear updated figures from metrics gurus Capers Jones and Howard Rubin. I suspect that while the mean (average) productivity/quality figures haven’t changed significantly, the standard deviation has increased. In other words, it’s likely that the best IT shops have gotten progressively better during the past decade, while the mediocre/incompetent IT shops have fallen further behind. Thus, some IT shops are paying attention to the latest developments in software engineering, and are profiting from it; others continue to ignore that advice, and are falling further behind.

In fact, if you take this argument one step further, you could argue that the debate is over, the battle is over, and the winners and losers have already been determined. I may well have been a decade ahead of myself when I predicted in 1992 (in Decline and Fall of the American Programmer) that the American programmer would go the way of the dodo bird by the end of the decade. But to an increasing extent, it appears that the “winners” in the software wars are the burgeoning software firms in India. It was reported a few years ago that 40% all the SEI-CMM level-4 and level-5 organizations in the world (which represents the “cream of the crop,” and the “cream of the cream of the crop,” in terms of software process maturity) are located in India. As for the IT organization in the U.S. that are still whining and complaining and finding reasons to explain why they still aren’t following common-sense software engineering practices: perhaps they’re already dead and just haven’t figured it out yet. The brain is dead, but the body is still twitching.

There’s something else to keep in mind: the scale of software development has changed drastically during the past decade. Remember: it was just a little over a decade that Microsoft released Windows 95. Even the most outspoken critic of Microsoft would have to agree that its current effort, Windows Vista, is staggeringly larger, sophisticated, and more complex. Similarly, Apple was supporting the System 6 and System 7 versions of its operating system in the mid-90s; now it supports a vastly more sophisticated (and vastly more reliable, and impervious to viruses) version of its OS X operating system. You can find similar order-of-magnitude differences between the vintage-1995 and vintage-2005 versions of Microsoft Office, Adobe Acrobat, Netscape Navigator, and other software products.

It’s not entirely clear that the marketplace really needs or wants all of the additional features, functions, complexity, and sophistication that the software product vendors have foisted upon us. But putting this issue aside, the important point is that the methods, processes, tools, and techniques that were adequate in the mid-1990s may not have scaled up to support the development of products 10-100 times larger and more complex. So maybe we have gotten better during the past decade, but just barely enough so that we can deliver more complex products in the same behind-schedule, over-budget, buggy fashion that we did back then. What was considered “good enough,” in terms of software features and functions, has obviously changed in the past decade; but for a lot of people, the threshold of “good enough” hasn’t changed very much in terms of software productivity and quality.

Aside from the problems of coping with increasing scale, what other reasons could U.S. software firms have had for not taking the necessary steps to improve their software engineering processes? Well, remember what was going on during the final half of the 1990s: the dot-com mania, and the urgent focus on Y2K remediation distracted a lot of firms from basic software improvement. As for the first half of the current decade: the dot-com crash and high-tech downturn forced many IT organizations to hunker down, and deal with sharply reduced budgets that make it difficult just to keep the lights on, with little or no extra time or resources to invest in process improvement.

But maybe it’s not all as negative and depressing as I’ve made it sound. After all, a lot of IT organizations have embraced extreme programming (XP) and “agile” development methods; that’s certainly an improvement over the rigid, inflexible waterfall methods of the mid-1990s. Similarly, the object-oriented methodologies that were just beginning to reach “mainstream” status in the mid-1990s have matured into second-generation versions of UML and RUP. And the software industry has finally begun to accept components, widgets, plug-in’s, and open API’s with a reasonable degree of enthusiasm.

On the other hand, an informal poll at our Albany conference today indicated that hardly anyone is using commercial estimating tools to generate non-superficial estimates of schedule, budget, and effort for today’s complex systems. Peopleware practices are still largely the same as they were a decade ago (and obviously far less generous than they were during the go-go years of the late 1990s); and as I’ve suggested in my recent blog posting, the politics and culture that leads to death-march projects is still much the same as it was a decade ago.

I expect that Howard Rubin, Capers Jones, and Herb Krasner will have additional insights and perspectives when we meet in New York City for another presentation of the Software Best Practices conference. Stay tuned…