My review of the literature and case studies (that have been published) indicates that prediction markets have improved the accuracy of forecasts, but the improvements have not been great enough to encourage widespread (or even minimal) acceptance. Furthermore, these studies like to average their results over a number of markets, disguising the fact that some markets improve forecasts, while others fail to do so. Some studies look at average absolute errors, covering up the fact that some predictions were underestimating the true outcome and others overestimating it. This means the real errors are as much as twice as large as those reported. Few, if any, explanations for the failures are ever presented. This raises the issue of consistency. In case studies such as these, where there is no clear under- or over-estimation tendency, for which a correction may be made, the prediction errors are just too great.
Clearly, if similar prediction markets do not provide consistently accurate forecasts, they will not be relied upon for any important business decisions.
This might be overstating my position. I don’t think there is much benefit to business at the present time. That isn’t to say that they will never be valuable in a business setting. I *do* believe that it is possible to make them work effectively, but they aren’t there yet (at least in the published cases).
My next paper (coming soon) will discuss this further (as well as the value of public prediction markets).
Paul
Chris, the title of the post isn’t exactly what I understand Paul is saying here.
I have a few issues trying to derive business benefits based on academic papers. Mainly because most of the use cases for enterprise prediction markets haven’t been well studied, though they’re reasonably well understood by those that work with them consistently, like Adam @ Inkling, Emile @ NewsFutures, Dave @ ConsensusPoint, Mat @ Crowdcast, etc. Adam in particular has been writing some great blog posts answering these exact kind of questions.
I guess it’s hard to have a common frame of reference to even talk about all this until everyone has seen (and preferably *ran*) an enterprise prediction market. As you say, accuracy isn’t the big ticket thing we should all be talking about when it comes to PM’s. The big benefits come from the “softer” assessments, such as communicating and collaborating within a company, sharing information, etc. (And yes, at a much faster rate.) While they don’t show up in academic literature as often as hard numbers, the soft reasons tend to be much more powerful.
I have to, respectfully, disagree with Jed and Chris with respect to accuracy. My point is that if we can’t rely on prediction market accuracy, why bother? Businesses will never adopt a new process for forecasting unless it offers a benefit (i.e. more accuracy, lesser cost, quicker results). The thing to keep in mind is that cheaper, faster will never be sufficient to replace more accuracy.
In my humble opinion, the “softer” benefits of prediction markets, while nice, cannot replace the usefulness of accuracy. If prediction markets are not consistently more accurate than other means of forecasting, they will never have a chance in the business world.
Again, I’m not saying that it is impossible to achieve this accuracy, I just think we have yet to prove it.
“This might be overstating my position.”
“Chris, the title of the post isn’t exactly what I understand Paul is saying here.”
I didn’t try to craft a title that would sum up Paul’s position, which is complex (since he is skeptical on the current PMs, but remains an optimist). I tried to craft a title that would fit the excerpt that I published. What I published is the conclusion he reached after assessing the prediction markets.
Jed uses OpenID, and I don’t have to approve his comments each time. Panos uses OpenId, and I had to approve his comments individually each time. I see that Paul uses OpenID, too, so I will see whether I will have to approve each his further comments individually. I yet have to understand fully OpenID, how to get one, and what are the benefits and usage.
“My point is that if we can’t rely on prediction market accuracy, why bother? ”
Yes, you are right.