All but my most loyal blog-fans (are there such people?) have probably forgotten that I received my copy of Dreaming in Code from Amazon back in mid-January, and began enthusiastically reviewing the preface and first chapter. I knew that I would soon be distracted by other assignments and deadlines, but I had hoped to finish the rest of the book on a long plane ride to Japan at the end of the month. Unfortunately, that didn’t happen: my assignments and deadlines followed me all the way over to Tokyo, and all the way home.

But I’ve picked the book up again, and I’m going to make another attempt to comment on the individual chapters — beginning with Chapter 1 (which is actually the second chapter, because author Scott Rosenberg thought it would be amusing to behave like a programmer and start numbering things at zero, rather than one). The chapter is titled “Doomed,” and it’s a multi-faceted discussion of the problems, and associated mood, of the Chandler software project in July 2003. For those of you who aren’t intimately familiar with the (black) art and (witch) craft of software development, this is an important chapter to read — for it eloquently describes how screwed up our software development profession really is. Can you imagine a project status meeting in any reputable field of engineering or science where the project manager sums up the situation, as Rosenberg does at the beginning of this chapter, by saying,

“John is doomed. He has five hundred hours of work scheduled between now and the next release … Katie’s doomed. She has way more hours than there are in the universe. Brian is majorly doomed. Plus he’s only half-time. Andy — Andy is the only one who doesn’t look doomed. There are no hundreds on his list.”

If nothing else, this should tell you that whenever you ask a programmer for an estimate of how long it will take him or her to write a program, from start to finish, what they’re really doing is guessing. How can you have a rational conversation with someone about the amount of work remaining to be done on a programming assignment if it requires “way more hours than there are in the universe”? How can a project manager have any hope of staying in control of a software development project when his instructions to his team are, “Everybody who has a list [of tasks they are things remaining to be done) with more time than there is in the universe needs to sit down with me and go over it.”?

Programmers are, by their very nature, hysterical optimists; and fortunately, many of the programer we’ve been creating during the past 50 years have been simple enough that we could overwhelm them by brute force. But ever since the hardware industry began giving us processors whose cycle time was measured in nanoseconds rather than milliseconds, and RAM memories measured in gigabytes rather than kilobytes, we’ve been dreaming of software that exceeds our intellectual capacity to create. And we’ve compounded the problem by gathering groups of dreamers together, and hoping they can collectively create something even more complex. In Rosenberg’s opening vignette, we’d be facing serious enough problems if it was only John who was “doomed”; but John’s work has to mesh with Katie’s, which is inextricably linked to Brian’s, which is interwoven with Andy’s.

Why is it so hard for these four people to make their individual creative software efforts work? (And if you think it’s hard with four, imagine what it’s like with four dozen, or four hundred — or, as is rumored to be the case, four thousand programmers who collectively created Microsoft Vista!) Well, to draw an analogy, imagine what it would have been like if you had assembled 3,500 architects, engineers, carpenters, electricians, plumbers, and construction workers together in mid-town Manhattan in June,1929, and said to them, “Hello, everyone! You’ve just been assigned to the Empire State Building project team! We’d appreciate it if you could all figure out how to work together, and create the tallest skyscraper in America! Oh, by the way, we’d like you to be finished with construction by March 1st of 1931! Good luck!”

Assuming that the assembled mob didn’t flee immediately, hopefully you would have heard someone call out, “Where’s the plan? Where’s the blueprint?” Actually, plans and blueprints did exist; and as I discussed in a blog entry last October, many of the details have been republished in a 77-page book of single-spaced typewritten notes called Building the Empire State.

But back in the Chandler project, as recounted in Dreaming in Code, things are different. John, whose title is “system architect,” says “There’s a bunch of reasons [why we’re behind]. In order to build something, you have to have a blueprint. And we don’t always have one. Then you hit unexpected problems. It’s hard to know how long something’s going to take until you know for sure you can build it.” Well, duh!

If you discovered that your next flight from New York to Chicago was taking place on a commercial airline whose flight-control software had been developed without a blueprint, I suspect you’d opt for a good old-fashioned train ride instead. If someone told you that the nuclear power system that provides electricity in your city was controlled by a software system developed without a blueprint, you’d probably leave town and head for a rural part of the country. Fortunately, these two types of systems, along with virtually every other form of “safety-critical” system, is developed with a blueprint. That doesn’t guarantee the elimination of all software defects, nor does it guarantee that the software development efforts will be finished on time, and within budget; but it does help explain why so much of the novel, creative software systems developed by the PC industry and the Internet/Web community is so wildly unpredictable.

Aside from this gloomy observation about the nature of software development, the remainder of this chapter wanders through a number of interesting aspects of the profession. We read about dragons and snakes, which are far scarier than old-fashioned “bugs,” because they are an “important problem that we don’t have consensus on how to attack.” We’re given a brief summary of the themes in one of the classic textbooks from the software field, first published in 1975: The Mythical Man-Month, by Fred Brooks (the link here is to the 1995 twentieth-anniversary edition, which has a number of additional observations by Brooks, and commentaries by various others, including yours truly, who probably know far less about software than Dr. Brooks). Aside from the book, Brooks is probably best known for the aphorism known as Brooks’ Law: adding manpower to a late software project just makes it later.

And we are introduced to the world of open-source software development. What’s important here is not the distinction between the “free” nature of open-source software products (e.g., Linux) versus “commercial” software products (e.g., Windows Vista, from Microsoft), but rather the radically different way of organizing the programmers who work on such projects. Rather than organizing the developers into the traditional hierarchy of programmers reporting to team leaders, department managers, divisional Vice Presidents, and other executive titles, the open-source approach is a flat network, a “cooperative group ethos built around a leadership style, like [Linux’s Linus Torvald’s], that encouraged newcomers, welcomed contributions, and [strives] to maximize the number of qualified participants.” In describing this approach, Rosenberg cites a small chunk of Eric Raymond’s “The Cathedral and the Bazaar,” which I urge you to read if you haven’t seen it yet.

One of the most important themes in “The Cathedral and the Bazaar,” as Scott Rosenberg points out, is something Raymond calls “Linus’s Law”: “Given a large enough beta-tester and codeveloper base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, ‘Given enough eyeballs, all bugs are shallow.’ I dub this: ‘Linus’s Law.’” And because of this, he argues that the network-style open-source approach to software development invalidates Brooks’ Law: “To Brooks’s Law I counterpropose the following: Provided the development coordinator has a communications medium at least as good as the Internet, and knows how to lead without coercion, many heads are inevitably better than one.”

And therein lay, apparently, a big part of the problem for the Chandler team whose efforts Rosenberg is describing in his book: “They [the Chandler team] were an odd hybrid: They functioned like an open source project in that they were posting their source code on the Internet and trying to build a community of volunteer developers around it; but they also felt and acted like a classic software start-up company, with a core group of programmers trying to get a new product off the ground and worrying about how long it was taking.”

And so the story of the Chandler project begins. It’s not clear to me whether anyone — including the development manager, Michael Toy; or Mitch Kapor, the founder and funder and mentor of Chandler’s corporate environment, Open Source Applications Foundation (OSAF); or John, Katie, Brian, Andy, and any other developers who joined the team — had given a great deal of deep thought to some of these underlying philosophies of software development. Well, actually, Mitch Kapor had thought about the philosophical issues; that much is evident from the comments that Scott Rosenberg makes in this chapter, but it’s not yet clear to me how Kapor and the others really intended to translate philosophy into practical decisions and activity.

All of this is great reading for anyone who thinks of software development as a personal career, or “profession”; alas, most of the “professional” software developers I’ve met throughout my career have never had the time, the energy, the patience, or the desire to do much thinking about what they were doing. This first chapter of Dreaming in Code also makes excellent reading for non-computer people in any walk of life who realize how much of their jobs, their day-to-day life, and the world they live in depends on the reasonably accurate functioning of computer systems, as well as the constant stream of innovation that makes today’s life what it is.

To put it another way, for the average “man on the street”: how on earth did we survive without Google? how did we survive without the World Wide Web, and browsers to surf the Web? What did we do before e-mail, SMS and IM messaging? How could today’s teenager survive without super-sophisticated cell phones, iPods, and various other gadgets — all of which would be useless hunks of plastic and metal, without the millions and millions of lines of code that make them do the wonderful things they do?

If we’re all so dependent on this stuff, doesn’t it make sense to spend some time learning about how this software stuff actually gets developed — particularly since it’s so unpredictable, unreliable, and uncontrollable? Some of us think so; and I can only hope that millions more will be inspired to do so after being exposed to Dreaming in Code.

Whew! All of that, in response to just one chapter of Scott Rosenberg’s book; indeed, I’m beginning to worry that my comments might be longer than the chapter itself. Well, I’ll try to keep my comments about subsequent chapters a lot shorter … and hopefully the review of Chapter 2 will appear tomorrow or the next day. Stay tuned…

Updated to add: