Prediction Markets: Trading Uncertainty for Collective Wisdom

4 Responses to Prediction Markets: Trading Uncertainty for Collective Wisdom

1. Bentley207B says:

   April 25, 2009 at 1:24 PM

   I’ve written a blog, commenting on the accuracy contention in Emile Servan-Schreiber’s paper. It can be read here:
   http://torontopm.wordpress.com/2009/04/25/judging-accuracy-in-prediction-markets/

   I still believe it is possible to obtain consistently accurate predictions, and I agree with the author’s contention that diversity is an important aspect. I also agree that prediction markets are more appropriately described as “betting exchanges” rather than prediction markets, but I don’t think the case has been made regarding accuracy in the field. I’d appreciate hearing your comments.

   I hope you haven’t lost any sleep over your upcoming “legal” troubles!

   Paul

   Log in to Reply
2. Chris F. Masse says:

   April 25, 2009 at 4:20 PM

   Paul,

   I will read your blog post and blog about it.

   PS: I have yet to answer your e-mail.

   PPS: There is no legal threat. There is just a prankster over there. The problem with that e-mailing list is that anyone with a Yahoo! e-mail address and a fake name can publish any crap.

   Log in to Reply
3. Emile Servan-Schreiber says:

   April 26, 2009 at 8:31 AM

   Hi Paul,

   There’s are simple reasons why the ten years-old HP study is still mentioned so often. It’s because it’s a landmark study and because it was designed as an experiment meant to be written about and published. In contrast, the vast majority of the prediction markets that are fielded in enterprise contexts are meant to create business value instead of published case studies.

   Nevertheless, it is our experience, and I would guess the experience of most other consultants in this field, that achieving *relative* accuracy (in comparison to other forecasting methods) even in degraded conditions (eg, few participants, low incentives, little management support), is the easy part. Only in rare cases, for instance when the participants are simply not knowledgeable, do you fail to get superior accuracy from the market.

   The hard part, the real hurdles to acceptance are (i) to get wide and consistent participation, (ii) to overcome the suspicion of decision makers who naturally don’t like to be second-guessed by subordinates, (iii) to get the organization to agree to ask important questions instead of trivial ones.

   Log in to Reply
4. Bentley207B says:

   April 26, 2009 at 9:23 AM

   Hi Emile…

   I couldn’t agree with you more about the “hard parts” that make it difficult to gain acceptance for enterprise prediction markets. And it is these hard parts that are crucial to generating the accuracy and efficiency in these types of markets that is necessary for acceptance in the corporate world.

   I understand that HP’s study was meant to be published, and there is a lot we can learn from it. Given that HP was attempting to forecast future sales, I would argue that they were meant to create business value. I still question whether the HP study really “proved” that the markets work well enough to be useful – even by the standard of relative accuracy. As my “proof”, I cited the still very significant errors that remain in the HP predictions and the fact that, even though some of them were “better” than the official forecasts, most were still not “better” enough to have made a difference in corporate decision-making. In accounting (my background), we use the concept of “materiality” to judge whether a figure (estimate) is “reasonably accurate”. An error is considered material if the magnitude is such that it would have resulted in a decision that is different from the one made based on the “accurate” estimate. It is a subjective test, requiring the use of judgement, but it is a good threshold for judging whether a prediction error is small enough to make the prediction “accurate” for decision-making purposes.

   It is a good paper, and I do believe that the accuracy of prediction markets can be proven. I just don’t think that this particular study shows it convincingly enough. Ensuring the “hard parts” are fulfilled will help provide the accuracy that is needed for widespread acceptance.

   Log in to Reply