I participated in a software “best practices” conference today in Jersey City, and listened to several speakers lament the mediocre state of software development today, with ad hoc development processes, lack of metrics, and various other poor practices leading to schedule slippages, budget overruns, and poor quality. Someone in the audience interrupted one of the speakers and remarked, “I think you could have identified the same list of mistakes and inadequacies in our profession 20 years ago. Why haven’t we gotten any better?”

This same conference has been held in Buffalo, Chicago, Washington, New York, and various other cities over the past few months, and this is the third or fourth time I’ve heard this question. It’s kind of discouraging to imagine that we haven’t improved the state of the art in the past 20 years; but after thinking about the situation for a while, I believe there are at least a dozen explanations:

1. Sturgeon’s Law (or, more precisely, Sturgeon’s Revelation) says that 90% of everything is crud; so perhaps we should just accept things for what they are.
2. On the other hand, perhaps software development, on average, has gotten better — but there are still lots of spectacular failures that get a lot of attention. If the only things that get attention in the computing journals and mainstream media are the failures, it’s easy to end up thinking that all of our projects are failures.
3. While average software development practices may have improved slightly over the past 20 years, it’s also possible that the standard deviation has increased dramatically — which means that the best software development organizations have gotten much better, while the worst software development organizations have gotten much worse (perhaps because they have less training or experience than before). Metrics guru Howard Rubin has frequently presented statistics documenting this phenomenon, though I haven’t seen any in the past couple of years.
4. We’ve gotten much more ambitious, and we’re developing much larger and more complex systems — which implies that if we’re still using the same informal, ad hoc development processes that worked reasonably well on simply systems 20 years ago, they would lead to more mediocre results today. At one point, I had some statistics about the massive growth in the code size of Microsoft’s word-processing product (i.e., Word) and Windows operating system throughout the 1990s, but I can’t seem to find it now … but I’m sure there are good examples out there somewhere.
5. There is little or no legal pressure to deliver higher-quality systems. Our mediocre software hasn’t killed many people, and hasn’t been identified as the direct cause of many (or any) Fortune 500 bankruptcies. Yes, there definitely have been software project failures that have cost someone (e.g., a customer, a vendor, or a group of taxpayers) tens of millions, or even a hundred million, dollars. But the problems are not perceived, by our politicians and lawmakers, to be sufficiently widespread and pervasive to lead to a software version of the Sarbanes-Oxley Act. There was a lot of discussion about the possibility of such regulatory oversight being imposed by the SEC if Y2K had led to more serious consequences … but at this point, hardly anyone remembers what Y2K was all about.
6. Competition in the software industry tends to be driven by featuritis (more functions, more bells and whistles), and/or time-to-market, not quality. One can argue that an emphasis on quality will make it possible to deliver working software more quickly, but it’s often more expedient to use an informal approach to deliver mediocre software much more quickly.
7. Individuals, managers, and corporate stock prices are geared toward short-term results, not long-term results. There are more promotions, bonuses, and other rewards if you build a software system very quickly, even if the long-term maintenance costs are much higher; taking a little bit longer to develop a software system with dramatically lower long-term maintenance costs is often harmful to one’s year-end performance review and bonus award. It should be emphasized, of course, that this phenomenon is not restricted to software development.
8. Compared to other engineering disciplines, ours is a very young field. Twenty years seems like a long time in the fast-moving computer field, but I suspect that it took several more decades, if not centuries, for various other engineering disciplines (e.g., building bridges or large office buildings) to show significant improvements. Even today, bridges sometimes fall down and buildings collapse; and even today, engineering projects suffer disastrous schedule delays and budget overruns. Boston’s Big Dig project, for example, was estimated to cost $2.5 billion in 1985, but had spent $14.6 billion as of 2006.
9. Software development is still, in essence, a human effort — and humans are not significantly smarter or dumber than they were 20 years ago; I say this after having accumulated somewhat more than 20 years of experience as a human. Our software development projects still suffer from overconfidence and naive estimates by inexperienced project managers; and large software development organizations have so much (human) inertia that it’s a wonder that they ever change.
10. We’ve got a generation gap: the people who developed software 20 years ago are no longer doing so, and many are reaching retirement age. They may or may not have attempted to pass on their accumulated knowledge and wisdom to today’s generation of Java/Ruby/AJAX developers, but the younger generation often refuses to listen. That’s not intended as a gratuitous insult: when I started programming in assembly language in the mid-1960s, I wasn’t interested in listening to war stories about the “good old days” of programming in raw machine language in the 1950s. Of course, as you get older, you gradually realize that the basic principles of software development and project management have remained largely unchanged, and that the technological differences of machine language, assembly language, COBOL, Visual Basic, Smalltalk, Java, and Ruby on Rails are largely irrelevant — but it takes a while to gain that perspective.
11. End-users are, at least in some cases, less tolerant than they were 20 years ago. When they first started using PC’s in the early 1980s, they were willing to reboot their machines several times a day, if one buggy program managed to not only clobber itself but also the operating system. Today, we expect to run a dozen different programs concurrently, and we’re highly offended if we have to re-boot more than once a week.
12. We still don’t get much support from senior management in large companies, when it comes to investing time, money, and other resources in making long-term improvements in software productivity and quality. And perhaps that’s because we still haven’t figured out how to explain the issues to these executives, as well as the complex tradeoffs between short-term costs and long-term benefits.

There may be more items to add to this list, and perhaps some of my dozen explanations can be disputed. Feel free to weigh in with your own opinions.