The tools basically consist of security features on top of open-source tools that are frequently used to build cloud-computing apps: Apache’s Hadoop distributed file system, Google’s Mapreduce and the University of Cambridge’s XEN Virtual Machine monitor. The research team at UT Dallas has focused on tools that will provide secure query processing, as well as tools that store sensitive data in encrypted format in order to add security to data storage devices.

The work is based on a project being carried out for the Air Force Office of Scientific Research, and additional steps are being planned to include the departments of Defense, Justice and Homeland Security, as well as various intelligence agencies and other companies. Demonstration of the tools is already under way at King’s College London and the University of Insubria in Italy.

It’s far too early to tell whether this particular set of tools will become widespread, commercialized, and “mainstream.” But the fact that the research and development is underway, and that it’s being sponsored and supported by such prestigious organizations, is promising. As the article says, “the biggest obstacle to wide adoption of cloud computing is concern about the security of sensitive data,” and the work underway at the UT Dallas Cyber Security Research Center encourages me to hope that we will have robust tools in the not-too-distant future, with which to build cloud-based systems that organizations and individuals can trust with their most sensitive data.

I have a somewhat unusual perspective on this issue, because in addition to my work as a project-management consultant in the IT industry, I also spend part of my time working as an expert-witness for attorneys, on computer-related project failures. Since we “experts” are typically brought into a case months or years after a lawsuit has commenced, and since the lawsuit typically doesn’t start until a project has collapsed, it means that the work we do is somewhat akin to archeology. Often, the key individuals who were there when the key decisions were being made have disappeared from the scene: they quit, retired, died, or were fired. Occasionally some of these folks are still around, and can be interviewed (though it’s only the folks on your side of the legal battle, because the legal system almost always precludes conversations with the key individuals on the other side, until formal depositions are taken, under oath, or testimony is heard at trial).

When you do talk to the project managers, or project stakeholders, or key technical people to find out what really took place when the project ran into trouble, it’s not surprising that memories are vague, inconsistent, and sometimes strongly biased. But the e-mail archives are typically still intact, and that’s the first thing the lawyers go after; in addition, there may be status reports, risk management plans, issue lists, Gantt charts from Microsoft Project, and various other documents that provide a more objective picture … and, just like archeologists, often you can brush away the dust of history, and see fairly clearly what key mistakes were made — and when, and by whom.

In these project management lawsuits — which typically involve disputes between the customer who originally asked for a system to be developed, and one or more vendors (and sub-contractors, systems integrators, and advisors) who promised to build that system within a certain budget and schedule — you quickly learn that the world is not completely black-and-white. Both sides made mistakes; neither side was perfect. The question usually is who made the biggest (most serious) mistake(s), when they were made, whether they were anticipated, whether they were “justified” (e.g., caused by events or forces beyond the control of the person or company that made the mistake), and what they did about the mistake.

Especially when you’re dealing with 20-20 hindsight, it also becomes evident that many of the mistakes are “venal,” in the sense that they might have caused some minor difficulty, but the project would have succeeded anyway. Maybe a task was finished a couple days late … but it wasn’t on the critical path, so who cares? Maybe there was a bug in the delivered software, and it remained unfixed for months after it first showed up on the “trouble report” filed by an end-user … but it was a cosmetic bug that made one of the user display screens look a little ugly, so who cares? Maybe the mistake added a thousand dollars to the cost of the delivered system … but if the total budget was three million dollars, then even though it was a larger mistake than I could have tolerated in my own personal checking account, who cares?

On the other hand, some mistakes are clearly cardinal mistakes; even if everything else went perfectly, that one mistake may have caused the failure of the entire project. Or they put the project at such extreme risk that the slightest additional problem will turn out to be the straw that breaks the camel’s back. Inevitably, things do go wrong in large, complex projects — despite our best intentions, and despite our best efforts — because (a) people are fallible, and (b) there are indeed events and forces beyond our control. So it’s not surprising that the largest “cardinal sin” that one finds in reviewing a project is the lack of a robust, up-to-date risk management plan. Sometimes it’s the lack of support (either in the form of benign neglect, or outright attempts to sabotage) for the project on the part of senior executives, or key stakeholders. Or it might be a deadline that was clearly impossible from the outset, or a project team that was utterly inexperienced and unfamiliar with the business domain of the project, or one of several other key categories.

But I’ve been talking about after-the-fact analyses here, i.e., attempts by a technical expert to look back into the past and determine what went wrong. For the confessor-priest in an IT project confessional, the situation is usually different: the mistake has just occurred, or it’s about to occur in the next few hours, days, or weeks. What then?

Well, the distinction between “venal” and “cardinal” is still relevant, and the confessor-priest can usually approach by asking the simple “So what?” question. The confessor priest can say to the project-manager sinner, “Okay, so you did X wrong, or you forgot to do Y, or you did Z when clearly you should not have done so. So what? Is the project doomed?” If not, then it may be necessary to apologize to someone, or to take some corrective action, but the most important advice from the confessor-priest is usually, “Get over it.” That is, do whatever you must to put it behind you, and move on. There’s more work to be done, more milestones to reach, and an “ultimate” deadline still looming in front of you. Brooding over past mistakes, and crying over spilt milk, won’t get you closer to the goal.

On the other hand, if the mistake did indeed fall into the “cardinal” category, it’s important to confront that reality, too. Even in this case, corrective action may be possible — but it will usually require major apologies, major effort and expenses, and an acceptance that promotions, bonuses, and other rewards will not be forthcoming.

In the worst case, the cardinal sin may be so bad that the project is effectively doomed. Nobody likes to hear that kind of assessment, and it’s often rejected as a “defeatist” attitude. Unfortunately, it’s sometimes combined with a bit of “martyr” behavior on the part of the project-manager sinner: he doesn’t want to come out and say it openly, but his post-mistake behavior basically says, “I really screwed up, and the project is doomed, but I’m going to continue working as hard as I can, so that I can show everyone I didn’t run away from my mistake.”

If it’s a one-person project, maybe that’s okay (as long as senior management and the business user knows about it). But when the project manager has a dozen (or a hundred, or a thousand) people working for him, then it’s really not fair at all. I’ve seen situations where the project manager made a critical mistake (or, more commonly, a series of critical mistakes that culminated in irretrievable disaster) but managed to keep it hidden for several months — while the members of the project team continued working at a frenzied pace, and while senior executives continued pouring money into what would eventually be identified as a bottomless pit.

All of which leads to an outcome that I haven’t mentioned up to this point, but which the confessor-priest needs to be prepared for: sometimes you don’t get fired by stakeholders and senior executives who don’t like what you’re saying. Sometimes the project-manager sinner clams up, gets stubborn, and refuses to talk to you any more. Whether the confessor-priest maintains his vow of confidential silence, or whether he decides, at that point, to take the bad news to senior management … well, that’s another ethical question to ponder. Anyone cast into the role of confessor-priest needs to make whatever decision he/she is comfortable with; all I can do is suggest that you think about it carefully in advance.

Tomorrow, we’ll turn to another aspect of the IT project confessional: resisting pressure from higher-level executives …

Assuming that the conversation takes place after a mistake has been made, the confessor-priest should first ask how recent it was. If the mistake was made within the past day or two, it’s possible that it can be corrected/fixed/recovered; more about that in a moment. But in any case, it will still be fresh in everyone’s mind, and it will be considered “relevant” to all concerned. By contrast, suppose the project manager says to the confessor-priest, “This has been troubling me for months, and I have to get it off my chest: six months ago, I gave my key software engineer a performance review without any salary increase at all, because I was too chicken to fight for it with my VP…” Chances are that (a) it’s too late to do anything about it, and (b) the VP won’t remember or care if you decide to bring the issue up now, in an attempt to rectify the situation.

Obviously, one of the main things that confessor-priest needs to figure out is just how “serious” the mistake is (or was). It’s one thing for the project manager to say, “I made a mistake, and our 12-month project is going to be a day late.” It’s something else entirely if the project manager says, “I lost my temper, yelled at the whole project team at the top of my voice, and called them a bunch of childish idiots. When I came into the office this morning, I discovered that every single one of them has quit and left town — and they erased every bit of technical work they did from day one of the project.”

Also, as suggested above, the confessor-priest needs to determine whether the mistake is “recoverable.” Quite a few project-management mistakes turn out to be “human” issues — inopportune statements, insults, jokes, or comments that may have offended a subordinate or a superior. Left to fester, the mistake could have grave consequences; but a timely and sincere apology can often undo the damage, and perhaps even lead to a better working relationship in the future. (I emphasize “sincere” here, because I’ve noticed an all-too-common tendency, perhaps made palatable by politicians, movie stars, and other public figures, to say something utterly outrageous in public, and then offer a bland, passive-voice pseudo-apology along the lines of, “I regret that X was said. It should not have happened….”)

Unfortunately, sometimes the mistake cannot be undone — at least, not with the resources at the disposal of the errant project manager, and not even with the assistance of the confessor-priest. If the entire project team really did quit, and if they’ve got better-paying jobs working for a competitor, it may not be possible to get them back again. If the project manager failed to carry out the required risk-management planning, and didn’t have a contingency plan when something went utterly wrong (a certain oil spill in the Gulf of Mexico comes to mind…), there may not be any practical way to plug the leak, repair the damage, and get things back on course.

All of this typically leads to one of four recommended actions on the part of the confessor priest, when the project-manager “sinner” describes the details of the mistake that he or she has made:

• ignore it — some mistakes really aren’t that bad. And some are “mistakes” only in the sense that they violate bureaucratic rules that had no meaningful impact on the project anyway
• fix it — it may cost money, it may take time, and it may take extra work — but many mistakes really can be rectified. Of course, the sooner you acknowledge it, and the sooner you ask for help and/or begin taking remedial action, the better off you are.
• ask for forgiveness, and vow never to do it again — if the mistake was a human-relations blunder of some kind (e.g., needlessly annoying/insulting a key member of the project team), it may be sufficient to grovel and beg for forgiveness. I find it amazing how rarely management-level people are willing to publicly (and sincerely!) acknowledge their mistakes; more often, they try to stonewall the situation, and bluff their way through it. But asking for forgiveness often works only once; the second time you tell your spouse that you’ve had an affair, and that you’re terribly sorry and really won’t do it again … well, it’s not likely to be very convincing.
• acknowledge defeat — hopefully it won’t happen very often, but let’s face it: sometimes you make a mistake that’s really serious, non-recoverable, and completely unacceptable no matter how much groveling and apologizing you do. If you were the Apple engineer that left the iPhone4 prototype behind in the bar a few months ago, chances are the best thing you could do would be to write a short, polite resignation letter and shove it under Steve Jobs’ door. But even here, sooner is better than later; and taking ownership/responsibility for the mistake (rather than making excuses, or trying to blame it on someone else) is the honorable thing to do.

Tomorrow, we’ll discuss what the IT confessor-priest should do if the project-manager sinner warns him of a mistake that he or she is tempted to commit, but has not yet actually committed…

But there’s an obvious analogy: suppose company X has hired me — just to use a silly example, because it’s easier for me to write in first-person form — as the IT project priest. An announcement is sent out, inviting troubled project managers to schedule a meeting with me. The date of my visit is publicized, and people are told that it’s okay to show up at the office where I’ve been sequestered, even if they haven’t formally scheduled a meeting.

So I show up, sit in my office with a nice cup of coffee and bran muffin, and catch up on my e-mail while waiting for someone to appear. Time passes, and I finish my e-mail; nobody has appeared. So I read the New York Times, spend a few minutes on the crossword puzzle before giving up in frustration, and reorganize my to-do list for the umpteenth time. Before I know it, the morning has slipped away; and after a quiet lunch alone (after all, nobody wants to be seen in public with the notorious “confessional priest”), I return to the empty office for an equally quiet afternoon.

What can we conclude from all of this?

If it’s a small IT organization, with only one or two project managers, maybe it means that nobody has any serious problems … at least, not now. Maybe there really are problems, but the project managers don’t know about them. Or maybe everything is actually on schedule, and all of the technical staff members, as well as the business users and key stakeholders, are perfectly happy. It may seem a little strange, but it’s not completely impossible.

In a large IT organization, it really is impossible. Well, maybe not completely impossible — but highly unlikely. Considering the industry statistics about how many IT projects are behind schedule, over budget, filled with bugs, and unable to meet user requirements, you would think there would be a long line of desperate IT managers lined up outside my office, hoping to receive either a miracle cure or a promise of forgiveness and absolution.

Also, why do you think I was brought here in the first place? Unlike “real” priests, I don’t operate as a non-profit charity, and I don’t rely on donations to pay the rent each month (nor do I live in a rent-free vicar’s mansion here in NYC). The decision to bring me in for this purpose is almost always made (and, more importantly, paid for) by a senior-level IT executive who knows that (a) lots of projects are in trouble, and (b) lots of project managers are frustrated, discouraged, and threatening to quit. That would make it even more likely that several people would be lined up outside my door (or sending me e-mail messages asking when they can meet with me).

But there’s actually something far more fundamental to consider: most project managers know when they’ve made a mistake, and when they’re in trouble. Maybe not the most junior managers, who are tackling their first project; and maybe not the most stubborn ones, who are determined to bull their way through any obstacle thrown in front of them. But most project managers are reasonably intelligent, and reasonably aware of what’s going on around them; and at least in the U.S. (though not necessarily in other countries), they operate in a culture where their subordinates will voice their opinions about how well (or poorly) the project is proceeding.

So an absence of people lining up to meet with the IT “confessor priest” usually means one thing: there is a strong atmosphere of fear and distrust in the organization, and none of the project managers are willing to take the risk that senior management will somehow find out they’ve reached out for help. In such an organization, there probably aren’t many people taking advantage of company-sponsored programs to help with problems of alcoholism, drug abuse, or marital problems. When it comes to project management problems, many IT organizations operate in a style not so different from the infamous “don’t ask, don’t tell” standard currently being debated in the U.S. military.

As indicated in a previous blog posting, the project confessional is supposed to be confidential. But if the “confessor priest” sits in an empty office at the end of the hall, far too many people can see the “sinner” project manager walking down that hall for a meeting. Meetings can be scheduled via e-mail, and can take place outside the office environment; but in some organizations, people are paranoid about their e-mail and text messages being intercepted. Whether real or not, the perception of such “Big Brother” oversight is enough to keep them away.

I don’t have any magical solutions for this kind of problem; all I can do is report to senior management that the level of fear and distrust is greater than they had imagined and/or acknowledged. A significant culture-change has to take place before the project-confessional concept can be put into practice, and that usually requires a different type of consulting engagement altogether.

Fortunately, this scenario is not very common. While I can’t promise absolute secrecy, it’s not too difficult to create enough privacy and confidentiality to satisfy most people. Meetings can be arranged via e-mail, but with the proviso that corporate e-mail be avoided. My cell phone number can be made available, and people can call me from someplace suitably private. And, in most organizations, people aren’t that terrified of making contact with me…

Indeed, think of it this way: if a project manager has made some mistakes, and if the project is in trouble, that fact is not likely to be a secret. Well, maybe it’s a secret today, because the project team and senior management have not yet become aware of the project manager’s blunder … but it’s only a matter of time. So, before things get completely out of control, and before the project manager has completely lost whatever options might be available to remedy the problem, he or she will often feel motivated to go find someone trustworthy they can talk to…

Of course, there’s always the possibility that the project manager’s mistake was really serious, or that it broke the law. We’ll discuss that in the next blog posting…

Stay tuned.

Before we delve into the more subtle issues associated with such a confessional, I want to cover the basics … and before I do that, I want to acknowledge that this is not some crazy idea that I thought up all by myself. The IT Project Confessional was one of many novel and innovative ideas developed over a period of several years, starting in the early 1990s, by a group of IT consultants and educators who gathered together in an unpaid, voluntary effort known as the “Airlie Council,” to offer suggestions and advice to the U.S. Department of Defense for improving their software procurement, acquisition, and development efforts; the group included such well-known names as Tom DeMarco, Capers Jones, Vic Basili, Susan Tinch Johnson, Frank McGrath, Tom McCabe, and others. Oh, yeah, and me.

The Airlie Council, formally known as the Software Program Manager’s Network (SPMN) operated through the 1990s, until its funding was cut off in 2001; some of its products and ideas have been preserved by a Washington-based consulting firm called American Systems, and can be found here on the Internet. Among the innovative ideas concocted by the group were such things as compiling a set of known “worst practices” to complement (and offset) the traditional “best practices” created in many organizations, as well as a “project breathalyzer” test to help make a quick assessment of the likelihood of an IT project running amok and failing catastrophically.

As for the project confessional concept: it became evident that the massive DoD organization had “politics” that were every bit as intense as what one would find almost anywhere else. If you were one of, say, 10 junior officers of approximately equal rank, and if you knew that roughly half of you might get promoted within the next year or so, chances that that you wouldn’t want to blurt openly about your mistakes, weaknesses, and screw-ups. And if your superior officer was a gruff, no-nonsense person whose management approach consisted mostly of yelling at people, or reassigning them to the U.S. equivalent of Siberia, then you probably wouldn’t want to tell such an officer that you had just made a serious mistake on a mission-critical IT project under your command.

And so the idea of a “confessional” evolved. The idea was that one of us would visit a military base or IT development organization where several projects were underway, and let it be known that we were available for “free” consulting about any project-management issues and problems that anyone wanted to talk about. Anyone who wanted to meet with us could contact us directly to schedule a time and place; most of the meetings were about an hour long, but they could be longer or shorter depending on the needs of the individual.

There was an agreement that our conversations were confidential, to encourage frank discussions. On the other hand, most of us (probably all of us) Airlie Council consultants had no military security clearance, so the project managers knew there was a boundary separating that which they could conceivably discuss with us, and that which was off limits. I mention this primarily because it also meant that certain “categories” of mistakes, or project decisions, were generally off-limits, and would not come up for discussion.

And I mention that point because even in a non-military/unclassified IT organization, there may be a boundary between things that the confessional “priest” can keep confidential, and things that he or she cannot. If a project manager says to me, “Forgive me, father, for I have sinned: I took a thousand dollars from petty cash to buy a plane ticket for my star programmer, so he could spend the weekend gambling in Las Vegas,” that obviously poses a serious ethical problem. We’ll come back to this point in a future blog posting…

In any case, the project-confessional meeting is usually over at the end of an hour or so; and there are several possible outcomes. Sometimes, the “priest” can offer immediate advice — even if it’s something unpleasantly negative like, “Sorry, kid, but I see no miracle cure for you; your project is doomed. Better update your resume, and try to be brave to tell your boss now that that you’ve made an irrecoverable mistake.” Of course, sometimes the immediate advice is positive, with some simple suggestions for solving the problem.

Of course, many real-world problems are not so simple that they can be solved on the spot, no matter how experienced the “priest” may be; thus, there may be an informal agreement that the “priest” will contemplate the situation for a day or two, and then communicate a brief recommendation by phone or secure e-mail. A followup meeting may be appropriate, especially if additional questions or discussion are required to understand the nuances of the problem; but this should not be considered the beginning of an ongoing, open-ended consulting relationship.

On the other hand, sometimes the best advice that the “priest” can offer is that some kind of ongoing consulting assistance is needed — perhaps to solve an ongoing technical problem, or perhaps to continue offering some management-related advice and guidance. To avoid conflict-of-interest problems, the “priest” will generally not recommend his own services; but any recommendation is likely to open a can of worms, since (a) it means additional fees and expenses will probably be involved, and (b) it almost certainly means that the details of the problem (i.e., the one that caused the “confessional” meeting in the first place), and the long-term solution to the problem, will become known to the “sinner’s” peers and superiors. Again, we’ll discuss this in more detail in a future blog posting.

So that’s the basic idea of a project confessional. It’s probably not the sort of thing you would be likely to see in a small IT organization with 10 programmers and 2 project managers. But it’s the kind of thing that could be extremely practical and helpful in a large IT organization with a few thousand technical staff members, a few hundred managers, and several dozen development projects underway at any given time.

More to come … stay tuned.

Or maybe it was something else; maybe something you forgot to do, some budget report you forgot to submit, some paperwork to keep the bureaucrats and bean-counters from making your team even more miserable than they already are. Whatever it is, it’s going to cost your project some precious resources, or add to the bureaucratic burden, or somehow put the project at much greater risk of delay or outright failure.

That being the case, wouldn’t it be great if you could find a quiet confessional booth somewhere, and whisper to a kind old priest inside the booth, “Forgive me, father, for I have sinned…”?

The problem faced by many of today’s IT managers is that they know they’ve made a mistake — but (a) it’s not obvious to them how they can undo, work-around, or rectify that mistake, and (b) there’s nobody they can talk to. For whatever reason, they feel that they can’t talk to their subordinates (after all, they’re the boss!), and they can’t talk to their fellow-manager peers … and most of all, they dare not confess their mistake to their boss. Why not? Because their boss would have a hysterical fit, or fire the errant project-manager on the spot; and his/her peers would sharpen their knives, and begin figuring out how to take advantage of the mistake when it comes time to award promotions, raises, and bonuses.

I don’t want to suggest that it’s like this for all companies; there must be a friendly, supportive IT organization out there somewhere. And it’s not an “all-or-nothing” situation: if you’ve made a mistake but know how to fix it, you can sometimes enlist the cooperation of your subordinates (“we’re in this together, guys, and I’ll really owe you one if you help me out). Indeed, you might be able to make an apologetic confession to your boss, if the mistake isn’t too expensive to fix …

But there are an awful lot of situations where that won’t work… and this series of blog postings is about the formal creation of an “IT Project Confessional” to provide a neutral, objective, confidential, no-risk (well, probably low-risk) mechanism for project managers to seek advice and guidance so they can recover from their mistakes and ultimately succeed with their projects.

As you can imagine, things are a little more difficult in the “real world” of an IT organization than they are in the priest’s confessional booth: you can’t just tell the sinner, “Repent, say three ‘hail Mary’s,’ vow to never commit such a sin again, and you will be forgiven.” Here are some of the topics I’ll be covering in the days ahead:

• Finding sinners – how do you get people to admit they might need help?
• Protecting the confidentiality of project managers discussing their mistakes
• Ethics: what if the project manager has violated a law or government regulation?
• Types of advice – should you tell the sinner to quit, work harder, confess publicly, or something else?
• Categories of project management sins: venal sins and cardinal sins
• Resisting pressure from higher-level executives who say to the confessional priest, “Off the record, no names mentioned, tell me what’s going on…”
• Forgiveness — is it possible? Practical?
• Anticipating a sin – what to do if the project manager says, “I haven’t sinned yet, but I know I’m about to…”
• Measuring results
• Providing follow-up references and resources for ongoing help
• Setting up a “Sinners Anonymous” for project managers who want to network and share their experiences with other sinners

Stay tuned … and if you know any project-management sinners out there, tell them to take a look, and offer their own ideas, experiences, and opinions…

But as an intellectual exercise, suppose for a moment that that was not true; suppose that we were doomed to continue using today’s hardware technology for the next 10 years. Would things get any better?

Those of us who have been working in the software industry for the past 40-50 years are likely to have a pessimistic reaction to such a scenario. “We’ve been writing lousy code, full of bugs, for the past 50 years — and we’ll probably continue to do so for the next 50 years. And we’ve been consistently over-budget and behind-schedule on our software development projects for the past 50 years — so why should we expect things to get any better in the next 50 years?”

Indeed, there is some cause for pessimism here, but there is also an optimistic response: “We have gotten better at project management, and we do write better software than we did 50 years ago. But as hardware technology has improved so astoundingly over the past several decades, business organizations and society, in general, have demanded that we solve bigger and more complex problems. And we have taken on bigger and more complex problems, until we reach the point that our efforts are just barely ‘good enough.’”

We could debate these pessimistic and optimistic perspectives for quite a long time, but for now we’ll put them aside. Back to the simple, straightforward question: what if we could only expect improvements in the computer industry from the “stuff” we do in software?

One thing is fairly obvious: it would take us another 5-10 years just to use the hardware we’ve already got! There are exceptions, of course, some of which we’ve hinted at in some of the previous postings in this blog thread. But particularly in the area of personal/home computers, and also in the area of small-business computers, we are using only a small fraction of the available computing resources. We could probably continue for another 5-10 years, writing the kind of “brute-force” software that lazy programmers have been able to write for the past decade or two; and at some point, we might have to start optimizing our code, and we might have to schedule the use of, and access to, our hardware resources. It would be extremely unpleasant to go back to the days of “batch scheduling” of computer jobs; and it would be unpleasant if we had to abandon the idea of dedicated, personal computing devices, and return to the days when lots of people had to share scarce hardware resources … but it could be done.

Again, let’s put that scenario aside for now. Let’s take a more positive, optimistic perspective: what kind of new, “good” things might we expect from the software industry? There’s one interesting perspective that we’ve been hearing from people for the past year or so: look how many apps are available on the iPhone! First it was 100,000 — which was pretty amazing — and then it was 150,000 and now it seems to be 200,000. Interestingly, I recall reading about numbers like this back in the late 1990s, when the Palm Pilot was all the rage. In a long-forgotten article in the Wall Street Journal (which I’m too lazy to track down on Google), I remember reading that Palm had cultivated a “cottage industry” of 40,000 developers who created tens of thousands of Palm apps, which they sold for a few dollars each. Of course, Palm didn’t have iTunes — and the mechanism for finding, purchasing, downloading, and installing the apps was time-consuming and tedious.

So now that we do have iTunes, does this mean we should expect a Moore’s-Law phenomenon with mobile apps? Will today’s figure of 200,000 apps escalate up to 2 million apps by 2015, and 20 million apps by 2020? Even if that were true (the likelihood of which I’ll discuss in a moment), it doesn’t necessarily mean that society will be fundamentally improved — if we had access to 2 million iPhone apps in 2015, it doesn’t necessarily follow that they would all be “killer apps.” In fact, only a small percentage of them would show up on everyone’s iPhone — but the rest could still serve a useful purpose, in the form of a “long tail” of “niche” apps that are only useful to a few people.

Indeed, the long-tail phenomenon is already quite visible in places like Amazon and the iTunes music store. In the old days, when music was sold on vinyl records, and books were only available on dead trees, retail outlets (e.g., book-stores and record stores) only had room for a limited inventory, and were thus constrained to stock and sell only their most popular items. But now, iTunes carries an inventory of approximately 5 million songs; and in any given fiscal quarter, almost every one of those 5 million songs sells at least one copy. And if it turns out that almost all of those songs sell only one copy, who cares? They’re all just bits on a disk drive, and it doesn’t really matter when you transmit one copy or a million copies to the customers who want them.

So maybe the software-driven future of technology is one in which we benefit from a long-tail phenomenon of millions of individual apps on the various hardware devices we have in our pockets, our purses, our homes, and our offices. But if that is the future, we need to ask: who will actually write all of those apps? People who already consider themselves programmers? Depending on whose estimate you believe, there may be a few million such individuals world-wide today, and if they spent their evenings and weekends working on clever apps (assuming that each of them had a clever idea), maybe that would get us from the current level of 200,000 iPhone apps to a couple million. Beyond that, I have my doubts…

Indeed, if we think that the most interesting version of the future consists of dramatically more application programs, we need to imagine a world in which people other than “professional” programmers could create them. Computer programming is no longer considered “rocket science,” so we don’t have to restrict our attention to people with university computer-science degrees; indeed, they don’t have to have any university degree. Even if we imagined that new apps required basic literacy and the educational equivalent of a high-school diploma, that would still be a very, very large number of people … indeed, perhaps enough to get us to the level of 20 million apps a decade from now.

To illustrate that this is not as crazy as it might sound, consider that until the mid-1970s, almost any kind of non-trivial computation for business, scientific, or engineering purposes required a programmer, who would hand-craft a program in FORTRAN to produce the required computational results. Today, we refer to such a program as a “spreadsheet,” and we refer to the “programmer” who creates it as a “person.” I don’t know how many copies of Excel have been sold by Microsoft, but I’m willing to be that it’s far more than 20 million; and there are many more than 20 million spreadsheets providing useful results.

So perhaps what we’re saying is that software-related technological advances will come from the introduction of a “tool” as simple and powerful as a spreadsheet, with which ordinary people can create whatever app they might imagine for their mobile computing device (or, for that matter, any computing device). Maybe it will be a “mashup” tool, developed along the lines of Yahoo Pipes, or the tools available from Microsoft and IBM and others. Maybe it will be some kind of widget, template, or other mechanism.

I don’t know if this is how the future will turn out, and I can think of one reason why there might be no relationship at all between the world of spreadsheet-developers, and the world of mobile-phone app developers. Consider this: spreadsheets are generally created in order to solve a “real” problem — e.g., because someone wants to add rows and columns of numbers in order to make a business decision. I’m sure there are exceptions, but most of us would not consider a spreadsheet to be “entertainment,” or “fun,” or some kind of “game.”

But if you look at the items in the iTunes App Store, the vast majority are games, fun, or somehow related to entertainment. There are, again, lots of exceptions (I’ve got only one game, Klondike, on my iPhone), and there are lots of apps that provide serious, productive solutions to some kind of problem. But you can’t get away from the fact that most of the apps are for amusement.

Not only that, they’re not for the amusement of the person who created the app. Instead, the app-developer is hoping to persuade others to play his game, and join in the entertainment. If millions of people are thus entertained, and if they’re willing to spend $0.99 to download and play the game, then the developer gets rich. Conversely, if nobody likes the game, or nobody even notices its existence, then the developer makes no money — and is likely to conclude that he wasted his time…

As for the “productive” apps that one finds on a mobile device, a large percentage are simply scaled-down versions of the same app that one already has on a laptop or desktop computer. In any case, though, here are the categories of apps that I think we’ll see a lot more of in the next several years, some of which may well turn out to be as much of a killer app as spreadsheets and Google:

• collaborative apps — it may be the next generation of Facebook, Twitter, and other “social media” apps, or perhaps something like the newly-released Google Wave
• virtual-world apps — most current versions of computer games could be thought of as a “virtual world” in which player(s) battle enemies, search for treasures, etc. But I think these will become far more sophisticated and intricate (e.g., like Second Life, with social rules and currencies), long-lasting, and multi-player in nature. If it can be embedded into your mobile device, with the ability to call, text, or email to you (and from you), it’s easy to imagine that some players could essentially abandon the “real world” altogether.
• more “location-aware” apps — indeed, the collaborative apps, and virtual-world apps, become all the more powerful if they know where they (and their associated human “owner”) are located, as well as knowing where other relevant participants/players are located. Looking at my iPhone, I see that roughly 25% of my apps are not only location-aware, but are able to use that information constructively. I think we’ve only begun to explore this area, because it’s only been the last couple of years that our mobile computing devices have had GPS mechanisms attached. The question to ask here, I think, is not what the “killer app” will be in the location-aware space, but rather who is likely to invent it…
• assistant/agent apps — aside from alarm clocks and simple “filters” that tell us if a designated keyword has been spotted in a newsfeed, most of today’s apps are fairly passive. We have to tell them that we want a task carried out, and we typically have to provide a great deal of specific data in order for the task to be performed. Tomorrow’s apps will gradually accumulate more and more “knowledge” about the humans they’re “connected” to, in terms of likes and dislikes, habits, areas of competence and expertise, short-term vs. long-term goals, willingness to make tradeoffs and compromises, etc. As a result, tomorrow’s apps will be able to take a more active role, offering advice, guidance, companionship, and warnings. Again, I have no idea what the specific “killer app” will be in this area; but it’s the generation of today’s children and teenagers that I think we should be watching for inspiration.

Even if my presentation is utterly boring, there will be free pizza and snacks … but apparently no free beer. Hey, you can’t have everything…

In a desperate attempt to keep my presentation from putting you to sleep within the first 30 seconds, I’m considering doing cartwheels across the room, and perhaps a hand-stand on the lectern. I’ve also included some video clips at unexpected spots in the presentation, which will hopefully draw a few giggles, snorts, and guffaws. And I’ve sent an email invitation to Madonna, asking her to join us for a cameo presentation … but she hasn’t responded thus far.

If you can’t join us in beautiful downtown Bedford for the event, you can download a PDF version of the presentation — though it doesn’t contain the video clips. Don’t complain: if you want the whole package, you gotta be there in person…

To download the 5.21 megabyte PDF file, click on the icon below, or click here.