Does wisdom require markets?
David Pennock January 19th, 2007
The Midas Oracle choir — I include myself here — preaches markets as the best way to tap wisdom in crowds. Markets force people to put money where their mouths are. Markets impose a Darwinism on accuracy. Markets are efficient. Markets beat pollsters and pundits at their own game. And so forth. But I’ll play devil’s advocate today (hot off the heels of playing heretic): do we really need markets?
Some of James Surowiecki’s most compelling examples of the wisdom of crowds involve something much simpler than running a market: averaging a bunch of people’s opinions. Whether the number of jellybeans in a jar or the weight of an ox, the average of a number of people’s guesses is (almost always) more accurate that (almost all) individual guesses.
A previous post featured a quote from an academic meta-study of over 200 studies on combining forecasts, which I repeat here:
Consider what we have learned about the combination of forecasts over the past twenty years… The results have been virtually unanimous: combining multiple forecasts leads to increased forecast accuracy. This has been the result whether the forecasts are judgmental or statistical, econometric or extrapolation. Furthermore, in many cases one can make dramatic performance improvements by simply averaging the forecasts.
Indeed, after many person-hours spent playing with probability judgments from the ProbabilitySports contest — trying all kinds of fancy expert-weighting schemes and machine learning algorithms — my colleagues and I found it exceedingly difficult to consistently produce a better forecast than by simply averaging everyone’s probabilities. Moreover, averaging came within striking distance two prediction markets — TradeSports and NewsFutures — performing a hair worse, but statistically insignificantly so.
This is consistent with a previous study where we averaged together the predictions of competitors in a Formula One racing prediction contest, obtaining remarkably accurate probability judgments.
Markets and betting suffer from an image problem and a learning curve — they appeal strongly to a certain demographic but are repelled by others. As much as I’d like this mentality to change, I don’t see it happening in the near future.
Emile Servan-Schreiber of NewsFutures has learned this first hand: many of his clients now prefer NewsFutures’s “competitive forecasting” system to classic prediction markets. The BRAINs at HP have also forgone markets in favor of calibrated scoring rules.
So am I jumping ship too? No. Despite the remarkable success of averaging, I believe prediction markets still tend to perform better when push comes to shove. The learning curve can and should be reduced through simpler user interfaces. Automated market makers are a great step on that direction, perhaps best exemplified at Inkling Markets. As for the image problem of markets and gambling — the legal, social, and political barriers — I have some thoughts but no good answers.
In order to get a fair assessment of the value of prediction markets compared to probability-averaging schemes, I think we ought to consider two functions separately: 1) expressing information that lies relatively ready-at-hand; and 2) stimulating investment in the costly discovery of information. See, e.g., the paper you co-authored, “Prediction Markets: Does Money Matter?” I’d suppose that merely averaging guesses would do pretty well at the former, but not so well at the latter. Your own work, as well as sound theory, suggests that to encourage research, we would probably prefer not just PMs, but real-money ones.
Besides MSR (Inkling and WSX), can you cite examples of popular, public probability judgment websites (other than ProbabilitySports)?? Thanks. Just curious. I want to see things on the public place, as opposed to behind walls.
Christopher Thompson:
No, wisdom does not require markets
http://www.midasoracle.org/200.....e-markets/
“Indeed, after many person-hours spent playing with probability judgments from the ProbabilitySports contest — trying all kinds of fancy expert-weighting schemes and machine learning algorithms — my colleagues and I found it exceedingly difficult to consistently produce a better forecast than by simply averaging everyone’s probabilities. Moreover, averaging came within striking distance two prediction markets — TradeSports and NewsFutures — performing a hair worse, but statistically insignificantly so.”
Except
A) These markets are biased by the larger sports betting markets. I’m guessing if you mapped their prices aganst the prices of the sharpest online books you would have similar numbers.
B) The larger “markets” are “inefficient” as to exploit public bias and increase the profits of the all knowing sportsbooks.
c) I can beat your “market” simply by playing the same probability as the market and deviating slightly in situations where I know there is bias.
In fact, empirical evidence strongly indicates to me that the average of true experts is clearly superior to the average of all. The *problem* with this, and the reason why it is not necessarily supported in the literature is that, most people aren’t really experts. And there is no easy way to identify “true experts” (So Average of True Experts (AOTE) is not an actionable strategy in most cases.
” trying all kinds of fancy expert-weighting schemes and machine learning algorithms” - Not surprising t
But, when markets are viable, I agree that *real money* markets should yield better results than “averaging”
Competitive forecasting would probably be better than internal prediction markets for all but the largest enterprises, just because even a semi-efficient market has its own high fixed costs (market makers, decent population of traders, maintaining liquidity blah blah blah). Prediction markets would be reign supreme in a public exchange setting, because that eliminates issues of trader population sizes and liquidity. But the capricious ruthlessness with which US regulations are enforced makes that impossibly risky.
I totally agree with Pennock’s view: We should be colored as biased when we propose “Markets for everything”.
Professor Bell comments that the use of markets is justified if the discovery of relevant information is costly.
I could add that the markets are able to aggregate only those information that have a cost of discovery which is lower than the market’s monetary incentive.
I think that the need for a framework on when to use markets is highlighted.
In an earlier work of mine ( http://gtziralis.googlepages.c.....EC2006.pdf ) I quoted Schrieber’s ( https://dspace.mit.edu/bitstream/1721.1/28514/1/57345537.pdf ) empirical formula:
Forecast_value = (Importance x Quality x Acceptance) / Effort
Schrieber is pointing to the right direction, we should use markets for forecasts of significant value, but further work is highly needed.
Alex has it right, and Tom Bell makes a good point although BRAIN, for example, does reward participants for performance. BRAIN actually doesn’t get a whole lot of respect in the PM world, but using some sort of calibrated averaging makes sense in corporate environments and where information isn’t very dispersed.
One reason why people might be suspicious of mechanisms like BRAIN is their relative opaqueness. There is the worry that administrators will be able to vary parameters to datamine whatever answer they’re looking for.
“opaqueness”
Exactly. There’s a strong philosophic argument for open markets, which empower traders, as opposed to mechanisms relying on secrecy and hierarchy.
—
Other links on the non-market Wisdom of Crowds:
How and When to Listen to the Crowd
http://www.overcomingbias.com/.....en_to.html
The wisdom of the ProbabilitySports crowd
http://blog.oddhead.com/2007/0.....rts-crowd/
The Wisdom Of Crowds: James Surowiecki on the predictive accuracy of the horse race betting markets
http://www.midasoracle.org/200.....g-markets/
“Require” and “need” force a “no” answer.
I more or less agree with Alex Forshaw above.
Opaqueness of non-market mechanisms probably looks good to managers who will be using the information produced (participants don’t see prices). I’d keep this in mind if I were selling “collective intelligence” expertise to corporations.
Shareholder activists on the other hand…
Real money markets open up the possibility of arbitrage of various markets to other markets. For instance, a certain political party may be bad/good for certain baskets of stocks, and an arbitrage could potentially be constructed. The same could work for hurricane markets vs. insurers.
To the extent arbitrageurs can link two markets together, I think they gain more “wisdom.”