Some of James Surowiecki‘s most compelling examples of the wisdom of crowds involve something much simpler than running a market: averaging a bunch of people’s opinions. Whether the number of jellybeans in a jar or the weight of an ox, the average of a number of people’s guesses is (almost always) more accurate that (almost all) individual guesses.

A previous post featured a quote from an academic meta-study of over 200 studies on combining forecasts, which I repeat here:

Consider what we have learned about the combination of forecasts over the past twenty years… The results have been virtually unanimous: combining multiple forecasts leads to increased forecast accuracy. This has been the result whether the forecasts are judgmental or statistical, econometric or extrapolation. Furthermore, in many cases one can make dramatic performance improvements by simply averaging the forecasts.

Indeed, after many person-hours spent playing with probability judgments from the ProbabilitySports contest — trying all kinds of fancy expert-weighting schemes and machine learning algorithms — my colleagues and I found it exceedingly difficult to consistently produce a better forecast than by simply averaging everyone’s probabilities. Moreover, averaging came within striking distance two prediction markets — TradeSports and NewsFutures — performing a hair worse, but statistically insignificantly so.

This is consistent with a previous study where we averaged together the predictions of competitors in a Formula One racing prediction contest, obtaining remarkably accurate probability judgments.

Markets and betting suffer from an image problem and a learning curve — they appeal strongly to a certain demographic but are repelled by others. As much as I’d like this mentality to change, I don’t see it happening in the near future.

Emile Servan-Schreiber of NewsFutures has learned this first hand: many of his clients now prefer NewsFutures’s “competitive forecasting” system to classic prediction markets. The BRAINs at HP have also forgone markets in favor of calibrated scoring rules.

So am I jumping ship too? No. Despite the remarkable success of averaging, I believe prediction markets still tend to perform better when push comes to shove. The learning curve can and should be reduced through simpler user interfaces. Automated market makers are a great step on that direction, perhaps best exemplified at Inkling Markets. As for the image problem of markets and gambling — the legal, social, and political barriers — I have some thoughts but no good answers.