In an earlier blog in this series, I reminisced about an experience I had had in a mainframe computer room, back in the mid-1960′s: a high-speed drum, roughly the size of a family-size washing machine, had begun vibrating so hard that it snapped the bolts anchoring it to the floor, and began vibrating its away across the computer room floor. It was quite something to watch — but what amazes me now is the memory that this heavy, expensive, state-of-the-art device had a storage capacity of only three megabytes.

Fast forward about 15 years, and I was agog at the notion that I could buy an IBM PC/XT clone for a few thousand dollars, with a 10-megabyte hard disk. Obviously, they’ve gotten bigger and bigger since then; I’m writing this blog on a desktop iMac that’s a year or two old, with 750 gigabytes of storage. If I got a brand-new machine. Out of curiosity, I just checked the online Apple computer store — which tells me that I could buy a new iMac with a 27-inch display, 2.8-gigaHertz quad-core CPU, 16 GB of RAM, and 2 terabytes of hard-disk storage for a total price of $3,449. Oh, yeah, they’ll throw in free shipping.

So, if Moore’s Law holds true in the area of storage capacity for the next decade or so, that means I can look forward to the possibility of a vintage-2015 iMac with a 20-terabyte hard disk, and a vintage-2020 iMac with a 200-terabyte disk.

Which raises the obvious question: what on earth would I do with all that storage? What would you do? What do we expect that most companies would do, assuming they could scale up their current “disk farms” by a factor of ten, or 100?

Interestingly, I’ve always been one of those people who kept needing and wanting more, more, more storage. These days, for example, I delete 90% of the photos I take with my digital camera — because otherwise my hard disk would soon overflow. Even with this kind of discipline, I’ve got 150 gigabytes of photographs from just the first five months of this year … plus digital photos, and scanned images of old-fashioned film photographs going back 40 years. I don’t know if I really need 20 terabytes of storage today, but I easily imagine needing it and wanting it five years from now.

On the other hand, a lot of competent computer-users that I know (a) have no idea how much disk storage capacity they have, and (b) have actually used less than 10% of their capacity, much of it with system files and applications.

I suspect that businesses are the same way: the larger, ambitious, technology-savvy companies (and government agencies, too) are constantly pushing the boundaries of available storage capacity. If you need an example, just think about Google for a few minutes. On the other hand, there are also lots and lots of companies (and government agencies) that probably don’t really know or care how much disk (or RAM, or tape or whatever) storage they’re using, and don’t feel an urgent need for more.

Indeed, what do we need all of this storage capacity for? Well, here’s one important clue: the reason I could live fairly happily with a 10-megabyte hard disk in the early 1980s is that I was storing only text-based files. I’m a fairly prolific author, but 10 million bytes of storage is a lot of writing! Obviously, what has changed since then is the desire/need to store digital music, digital images, and video. And while we were once content to capture, store, and manipulate images consisting of 480-by-640 pixels, now we’re dealing with images that often require 10-15 megapixels each.

Throw in maps, satellite images, and a constant stream of still and video images from hundreds (maybe thousands) of security cameras all around us, and it’s not too difficult to see the possibility that we’ll continue using up the increased storage capacity brought to us by the wonders of Moore’s Law … at least for another 5-10 years. If we decide that we want our video cameras to capture not just 10 frames per second, but 100 frames, or even 1,000 frames per second, that will require even more storage. (Why we would want 1,000 frames per second for anything other than a few specialized applications is beyond my ability to comprehend, but that may simply reflect a lack of imagination on my part.)

If we eventually reach the point where we have no more music, graphic, or video data to record, what else require massive amounts of storage? There are probably several answers to that question, but here’s one possibility: biometric data. If we could monitor, record, and manipulate all of the data associated with each heart-beat, each breath we take, each hiccup and burp, each movement of major muscles, each twitch of an eye, each … well, you get the picture. There is an enormous amount of data that we could record about every human on this planet, from the moment we’re born until the moment we die.

There are three important consequences of such a scenario — i.e., the scenario where we’re storing 10 times, or 100 times, as much data as we already store:

• Data mining will become increasingly important, because patterns and trends will be more important than the individual data items themselves. Well, some data items may be extremely important (e.g., the heartbeats shortly before and after a heart attack, or the video images captured by a security camera just before and after a terrorist incident), but the vast majority of the data elements will be mundane, boring, and ultimately irrelevant.
• Data analysis will also become important, which in turn will place a greater emphasis on the CPU “horsepower” needed to analyze, manipulate, and “crunch” all of the data we record and store.
• Data visualization will become more important, too. Good data mining might help us extract the tiny subset of data that we really do want to look at; but we’re also going to need new ways to visually absorb and digest massive amounts of potentially relevant data. The work of people like Edward Tufte will gain even more respect; and the work of innovative groups like Digg Labs (you’ve got to check out their work!) will also gain more attention.

Bottom line: storage capacity is likely to benefit as much from Moore’s Law as the related “dimensions” of cost, speed/power, and size/footprint. Exactly what we’ll do with vastly greater storage capacity isn’t completely clear, but I’m confident that it will help us use computers in qualitatively different ways 5-10 years from now.