I first noticed a blog article about Yahoo Pipes a couple weeks ago on the O’Reilly Radar blog, but didn’t have time to read it at all. That was followed by roughly four more increasingly-enthusiastic O’Reilly blogs (a list of which I’ll provide below), and an equally enthusiastic article on Steve Borsch’s Connecting the Dots blog; it also got picked up on EDS’s Next Big Thing blog. I’ve now skimmed some of the material — for a total of about 15 minutes — and I think it’s important enough to draw your attention to it; but Yahoo still hasn’t posted much documentation.

In a nutshell, Yahoo Pipes is a free, Yahoo-provided online service that lets anyone use a visual editor (for programmers: think CASE or IDE tools; for non-techies, think Visio) to combine data from several popular “feeds” (e.g., new articles from The New York Times, or photos from Flickr, or lists of available apartments from CraigsList) and combine them to create “data mashups.” As the data is combined, it can be sorted, counted, filtered, and manipulated in a variety of other useful ways, much of which is reminiscent of the UNIX “pipes” concept that dates back to the 1970s. The resulting output can be something quite unique; for example, take a look at “New York Times Thru Flickr,” a pipe created by Daniel Raffel to take the New York Times homepage, passes it thru Content Analysis and uses the keywords to find Photos at Flickr. Why anyone would want to do this is open for debate, but the results are pretty stunning.

As the Yahoo Pipes home page explains,

“You make a Pipe by dragging pre-configured modules onto a canvas and wiring them together in the Pipes Editor. Each Pipe consists of two or more modules, each of which performs a single, specific task. For example, the Fetch module will retrieve a feed URL, while the Sort module will re-order a feed based on criteria you provide. Each module has one or more terminals, represented by small circles in the interface. You can wire modules together by clicking on one module’s output terminal and another module’s input terminal. Now the output from the first module will serve as input to the second module….”

Well, that’s not enough for me to become an instant Pipe-creator (or, as we’ll perhaps start calling them, a plumber), but that may well because I’m just a little rusty. I have a feeling that the learning curve will be an hour or two, after which “Eureka!” light-bulb will go off, and I’ll internalize it to the point where I won’t be able to remember the day when I didn’t know how to do it. For those who spend their days working with the details of HTML, XML, RSS, JSON, and such things, the internalization process may well be instantaneous.

But it remains to be seen how long it will take for the “amateur” to learn how to create something useful and interesting with Yahoo Pipes? Of course, you don’t need to know HTML to build a simple Web site these days; with visual, drag-and-drop tools like Apple’s iWeb, it’s almost trivial to create a fairly decent-looking Web page in a matter of minutes; and people are busily uploading amazingly creative stuff to Flickr and YouTube with no technical skills at all. At first glance, it looks like the drag-and-drop nature of Yahoo Pipes’ visual editor might be equally straightforward … except that you have to type in at least a modest amount of geeky stuff to specify the details of the various data feeds and filtering rules that you want to impose. It may turn out that Yahoo Pipes becomes a plaything of hackers and geeks; it will be interesting to see how things develop.

Also, will Yahoo Pipes be used to create simple, one-off applications for personal use — e.g., much the same way that the overwhelming majority of Excel spreadsheets are relatively simple creations used only by the person who developed them in the first place. Things get a lot tougher when you create an Excel-based “application” to be used by a wide spectrum of people throughout a department or division; security, error-checking, ease of use, documentation, and performance start to matter. Obviously, the same issues are relevant for Yahoo Pipes; my instinct is that it will be a tool for ad hoc, low-volume, individual use … but I may be wrong.

Anyway, here’s the list of relevant blog articles from the O’Reilly site:

• February 7: “Pipes and Filters for the Internet“
• February 7: “Yahoo! Pipes: THe Modules for Building Pipes“
• February 8: “Yahoo! Pipes: Deconstructing a Pipe“
• February 8: “Yahoo! Pipes: Some Tips“
• February 12: “Early History of the Web Pipes Concept“

The interesting thing is that if Yahoo Pipes does become popular, it’s virtually certain that other vendors will quickly jump on the bandwagon. Indeed, I wouldn’t be surprised if Google is already hard at work on a similar concept; and for all I know, traditional players like Apple, Microsoft, and IBM might be doing something too. It could generate a niche sub-industry of startups and tiny “pipe-builders” who attach their work to these large creations.

Or it could all fade away …

But it all looks so incredibly clever and cool that I very much hope it succeeds. And somehow, during the coming weeks and months, I plan to squeeze out some extra time to become proficient with Yahoo’s tools, and also to watch the collection of publicly-accessible pipes — of which over a thousand are already listed on the Yahoo site, though the vast majority appear not to have been accessed or used by anyone else.

If you are doing something interesting in this area, or if you become aware of any “practical” applications (or, for that matter, something that just makes you gasp with awe and amazement), let me know.