Trust in the wisdom of the crowd … if you can figure out what it is

Bookmark and Share

December 31st, 2006

This is the time of year when people look back on the year that’s ending, and forward to the year ahead. What did 2006 really mean, and what’s in store for us in 2007?

I’ve read lots of different assessments and forecasts, and have been impressed by their variety — as well as the confidence the pundits have in their own opinions. But I was particularly struck by a news headline on the Internet this morning, which proclaimed, “Poll: Americans see gloom, doom in 2007.” This popped up on Yahoo News, which has the annoying habit of deleting many of its stories after a few days or weeks. In case the hyperlink doesn’t work when you get around to reading this commentary, let me summarize what the article says about the results of a December 12-14 Associated Press-AOL News poll:

  • 60% of respondents think the U.S. will be the victim of another terrorist attack in 2007.
  • 60% think “it is likely that bad guys will unleash a biological or nuclear weapon elsewhere in the world.”
  • 70% of Americans predict another major natural disaster (e.g., another Katrina) within the U.S.
  • 70% expect worsening global warming.
  • Only 29% think it is likely that the U.S. will withdraw its troops from Iraq.
  • 35% of Americans predict the military draft will be reinstated.
  • 25% anticipate the second coming of Jesus Christ.

Wow — those are fairly sobering predictions, especially if one believes that the predictions have a significant chance of being accurate. Of course, if 25% anticipate the second coming of Jesus Christ, that presumably means that the remaining 75% of the population does not … so maybe we can leave that alone. But if 60% of the population thinks the U.S. will be the victim of a terrorist attack, should we take that seriously — and do something about it? If 70% of Americans predict another major natural disaster within the U.S. next year, should we evacuate Florida and New Orleans? Or should we just ignore the predictions? After all, what scientific basis could these folks have for a prediction about a major natural disaster? It’s unlikely that very many of them are climatologists or weather reporters.

Wisdom of CrowdsIn my April 28th blog posting (which I’m sure you remember, but which you can find here if you need to re-read it), I commented on James Surowieki’s book, The Wisdom of Crowds, which discusses numerous experiments demonstrating that groups of people did far better than the average individual in various areas of estimating and decision-making. I don’t know whether Mr. Surowieki would extend his endorsement of crowd-wisdom to predictions of terrorist attacks and major disasters, but it made me feel that perhaps I should do a little more research before casually dismissing these predictions.

My first instinct was to track down the details of the polls itself, rather than simply trusting the condensed summary in the Yahoo News article. After all, who has even heard of “AOL News,” which purportedly conducted the poll in cooperation with the Associated Press? So I googled the phrase “Associated Press AOL News poll,” and got a hodge-podge of 1.3 million results, the very first of which was a link to a slightly longer version of the same article in Salon, which also informed me that:

  • 19% of Americans think scientists are likely to find evidence of extraterrestial life.
  • 35% predict that a cure for cancer will be found.
  • 80% predict that lawmakers will raise the current federal minimum wage.
  • 57% believe it is likely that another state will legalize gay marriage.

I also learned that the poll was conducted by Ipsos, a well-respected polling firm, not by AOL or Associated Press itself.

Okay, so that’s fine — but the second item in the Google search results was a link to an article that appeared on the same day in ABC News, entitled “AP Poll: Americans Optimistic for 2007.” It turns out that the article was describing the same poll, taken by the same Ipsos polling firm — but instead of all the detailed items reported by AP and AOL News, it tells us that:

  • 75% believe that 2006 was a good year for themselves and their families.
  • 72% feel good about what 2007 will bring for the country.
  • 89% are optimistic about the new year for themselves and their families.

Huh? How can 72% feel good about the 2007 prospects for the country, and how can 89% feel optimistic for themselves and their families in the new year, if a preponderance (60% and 70%, respectively) also believe that there will be another terrorist attack and a major natural disaster? Those percentages are too high to allow for the possibility of disjoint sets — i.e., entirely separate groups of feel-good optimists, who are independent of, and unrelated to, the terrorist-threat weather-worriers. By contrast, it’s theoretically possible that the 19% who believe scientists will find evidence of extraterrestial life are entirely disjoint from the 25% who anticipate the second coming of Jesus Christ.

It would really be interesting to start doing some correlations — e.g., how many people who believe in X also believed in Y? How many who believe in X and Y also believe in Z? But there are dozens, if not hundreds, of such correlations one might explore, and it’s unlikely that we’ll ever see them reported in mainstream-media articles. If the polling firm would make the raw data available, as a free service to all of us, then we could carry out our own analyses; but I very much doubt that Ipsos will ever do that. By the way, it looks like the details of the results from individual questions in the Ipsos polls can be displayed (but not downloaded) as a PDF file by selecting the desired poll from a list on this page; but as far as I can tell, the poll discussed in the Salon, ABC News, and AP-AOL News articles have not been uploaded yet. If you’re interested, you might wait for a few days, and take another look at the Ipsos page.

Just to round things out, AOL News reported in this article that the same Ipsos survey found that George Bush was the most commonly identified “villain of the year” (outdrawing Osama Bin Laden and Saddam Hussein!), but was also the most commonly identified “hero of the year.” And Fox News couldn’t resist telling us, in this article, that 81% of Americans believe in angels, 18% do not so believe, and one percent just aren’t sure.

Unfortunately, my conclusion about all of this is that we don’t know a damn thing about the wisdom of this crowd of roughly 1,000 people who participated in the poll. What we’ve learned, I think, is that you can get almost any results you want if you ask enough questions. It’s too bad, because I think that crowds could be asked some focused, relevant questions about topics for which they might be expected to have “informed” opinions, and we’d be able to learn when and where it makes sense to “listen” to the crowd’s opinion, and when/where we should ignore it. I had high hopes that an Internet-based trend-identification mechanism that I discussed in my April 28th blog entry — something called Future Monitor – might be able to contribute something in this area, but I just took another look at the site, and it appears to be largely dormant.

As for me, I’m still not sure about those angels. But ask me again next year, and perhaps I will have made up my mind.

Leave a Reply