Midas Oracle .ORG » accuracy

What has been the best InTrade prediction market ever? Has the ObamaCare prediction market at InTrade been “ahead of the commentary”?

Chris F. Masse — Tue, 16 Mar 2010 16:05:54 +0000

Jason Ruspini (who feels that the health care reform proposal might well be adopted) wanna feedback from you, folks.

Which InTrade prediction market(s) has/have been ahead of the Press (rather than the other way around)? What is/are the best (most divergent from the commentary, and correct) InTrade prediction market(s) in people’s memories?
Do you sense that the ObamaCare prediction market at InTrade fits these 2 criteria?

UPDATE: I asked The Brain whether he meant generalist media or political media, and he meant “generalist”. That makes all the difference in the world.

ADDENDUM

More info on health care reform on Memeorandum.

Previously: Insider trading in the InTrade prediction market on health care reform?

Midas Oracle = About + Archives + Authors + Best + Contact + Information Technology + Links + Probabilistic Predictions + Post-Mortem Predictions + Prediction Exchanges + Prediction Software + Publish Your Post + How To Publish

Truth in Advertising – Meet Prediction Markets

Paul Hewitt — Sun, 14 Mar 2010 21:54:53 +0000

Most published papers on prediction markets (there aren’t many) paint a wildly rosy picture of their accuracy. Perhaps it is because many of these papers are written by researchers having affiliations with prediction market vendors.

Robin Hanson is Chief Scientist at Consensus Point. I like his ideas about combinatorial markets and market scoring rules, but I think he over-sells the accuracy and usefulness of prediction markets. His concept of Futarchy is an extreme example of this. Robin loves to cite HP’s prediction markets in his presentations. Emile Servan-Schreiber (Newsfutures) is mostly level-headed but still a big fan of prediction markets. Crowdcast’s Chief Scientist is Leslie Fine; their Board of Advisors includes Justin Wolfers and Andrew McAfee. Leslie seems to have a more practical understanding than most, as evidenced by this response to the types of questions that Crowdcast’s prediction markets can answer well: “Questions whose outcomes will be knowable in three months to a year and where there is very dispersed knowledge in your organization tend to do well.” She gets it that prediction markets aren’t all things to all people.

An Honest Paper

To some extent, all of the researchers over-sell the accuracy and the range of useful questions that may be answered by prediction markets. So, it is refreshing to find an honest article written about the accuracy of prediction markets. Not too long ago, Sharad Goel, Daniel M. Reeves, Duncan J. Watts, David M. Pennock published Prediction Without Markets. They compared prediction markets with alternative forecasting methods for three types of public prediction markets: Football and baseball games and movie box office receipts.

They found that prediction markets were just slightly more accurate than alternative methods of forecasting. As an added bonus, these researchers considered the issue that prediction market accuracy should be judged by its effect on decision-making. So few researchers have done this! A very small improvement in accuracy is not considered material (significant), if it doesn’t change the decision that is made with the forecast. It’s a well-established concept in public auditing, when deciding whether an error is significant and requires correction. I have discussed this concept before.

While they acknowledge that prediction markets may have a distinct advantage over other forecasting methods, in that they can be updated much more quickly and at little additional cost, they rightly suggest that most business applications have little need for instantaneously updated forecasts. Overall, they conclude that “simple methods of aggregating individual forecasts often work reasonably well relative to more complex combinations (of methods).”

For Extra Credit

When we compare things, it is usually so that we can select the best option. In the case of prediction markets it is not a safe assumption that the choices are mutually exclusive. Especially in enterprise applications, prediction markets are heavily dependent on the alternative information aggregation methods as a primary source of market information. Of course, there are other sources of information and the markets are expected to minimize bias to generate more accurate predictions.

In the infamous HP prediction markets, the forecasts were eerily close to the company’s internal forecasts. It wasn’t difficult to see why. The same people were involved with both predictions! The General Mills prediction markets showed similar correlations, even when only some of the participants were common to both methods. The implication of these cases is that you cannot replace the existing forecasting system with a prediction market and expect the results to be as accurate. The two (or more) methods work together.

Not only do most researchers (Pennock et al, excepted) recommend adoption of prediction markets, based on insignificant improvements in accuracy, they fail to consider the effect (or lack thereof) on decision-making in their cost/benefit analysis. Even if some do the cost/benefit math, they don’t do it right.

Where a prediction market is dependent on other forecasting methods, the marginal cost is the total cost of running the market. There is no credit for eliminating the cost of alternative forecasting methods. The marginal benefit is that expected by choosing a different course of action than the one that would have been taken based on a less accurate prediction. That is, a slight improvement in prediction accuracy that does not change the course of action has no marginal benefit.

Using this approach, a prediction market that is only “slightly” more accurate, than those from alternative forecasting approaches, is just not good enough. So far, there is little, if any, evidence that prediction markets are anything more than “slightly” better than existing methods. Still, most of our respected researchers continue to tout prediction markets. Even a technology guru like Andrew McAfee doesn’t get it , in this little PR piece he wrote, shortly after joining Crowdcast’s Board of Advisors.

Is it a big snow job or just wishful thinking?

[Cross-posted from Toronto Prediction Market Blog]

Why you should never trust David Pennock when he brags about the accuracy of his predictions

Chris F. Masse — Sun, 14 Mar 2010 10:22:04 +0000

He cherry-picks positive outcomes in hindsight, and he measures probabilistic predictions categorically and in isolation.

Don’t be fooled by any “research scientist”.

OSCARS 2010: Did Justin Wolfers brag too much and too loudly? — [RELATIVE ACCURACY DEPARTMENT]

Chris F. Masse — Wed, 10 Mar 2010 18:47:16 +0000

Jason (a Freakonomics reader):

You are giving yourself WAY too much credit. Siskel and Ebert successfully predict these awards 100% year after year. This isn’t a difficult thing to predict. Predicting something like the NCAA tourney, that would be an accomplishment, but if you look at rankings and your prediction market, you will fail just as much as the average bracket.
— Jason

Ray Kurzweil (Robin Hanson’s guru) got his 1999-2009 predictions … all wrong.

Chris F. Masse — Wed, 06 Jan 2010 10:01:08 +0000

- Michael Anissimov: Kurzweil’s Failed 2009 Predictions

- More at the Next Big Future — (an excellent blog)

UPDATE

How the prediction market industry lied to the media (and the public, YOU) about Hewlett-Packard’s pilot tests regarding enterprise prediction markets

Chris F. Masse — Fri, 04 Dec 2009 14:52:49 +0000

Paul Hewitt on the Hewlett-Packard (HP) prediction markets:

Another tried-and-true method is to repeat a statement over-and-over until it becomes a “fact”. The best example of this, regarding prediction markets, is the claim that they are “proven” by virtue of HP’s remarkably lop-sided, unmistakable, trouncing of internal forecasts by a score of 6 – 2. It is usually cited as “6 out of 8″ or 75% better than the official forecasts. The “truth” is that the “wins” were by minuscule margins and many of the forecasts were equally bad. [...]

Read the rest, here.

External Link: Paul Hewitt’s blog on prediction markets

Andrew McAfee’s anecdote on enterprise prediction markets is a load of bullsh*t.

Chris F. Masse — Thu, 03 Dec 2009 00:20:05 +0000

Paul Hewitt:

I’m curious why Andrew McAfee would cite this example in support of prediction markets. Though he mentions that his executive students had conducted a “simple poll”, in fact, they really created a pari-mutuel market, which provided the “risk and gain”. It was not a prediction market, as the executives believed it was. There was no buying and selling of shares. They simply purchased their “horses” (time slots). It might be more appropriate to call the entries “tickets” rather than “shares” and “bets” rather than “investments”.

The “consensus” ticket was correct. No surprise here (favourite), and the payoff was slightly higher than it should have been, because of the slight long shot bias. Andrew might have corrected them on their terminology, too. Prediction markets are not run to obtain a *consensus* of opinion. They are run to *aggregate* information from the market participants.

It really would have been interesting to sit in on the class, had the fellow arrived at one of the other time periods. Would the executives have cheered that their “prediction market” had worked like a charm? Surely, someone would have jumped up and said “but it’s well-calibrated!” Had the wrong “horse” won, would *any* of the executives have been confident enough to present a proposal on the use of prediction markets in their companies? I’ll bet all of them would have been a bit concerned that the front runner didn’t win.

Andrew writes that it showed the “power of incentives”. Actually, it did not, at least in the sense we usually define “incentives” in prediction markets. In a true [prediction market], the incentive is presented when the market price is not “accurate”. The informed trader either knows this or has an incentive to obtain more information and set the price right. His incentive is the profit from trading to correct an inaccurate market price. There was no change in the price of any of the “shares” in the market they described. Hence, there was no “incentive” in this sense.

Andrew also states that this experiment shows how easy it is to set up “convincing demonstrations” of prediction markets. I agree, they are easy to set up (at least technically). Making them work is much, much more difficult.

I’ve read Andrew McAfee’s other posts on prediction markets, and it appears he’s been slipped a bit too much of the PM Kool-Aid that is going around. If you don’t read the prediction market research papers with a critical eye, you can get caught up in the over-enthusiastic reporting of results and come to the conclusion that prediction markets are a *proven* technology for greatly improving corporate forecasting problems. Nothing could be further from the truth at this point. Worse, there is not much useful research being published.

I think Andrew McAfee should have presented his students with a more balanced summary of the strengths and weaknesses of prediction markets.

That’s my take on it. However, I am happy that the executives were interested in prediction markets! There is some hope, yet.

Accuracy of millenium predictions

Chris F. Masse — Wed, 02 Dec 2009 22:00:16 +0000

Millennium Predictions That Got Things Right – and Those That Didn’t.

Robin, just because we don’t have another method of accurately predicting an outcome doesn’t mean we have to be so appreciative when a prediction market comes up with a forecast just before the outcome is revealed.

Chris F. Masse — Thu, 26 Nov 2009 08:56:01 +0000

- First missile —just an appetizer.

- Second missile: A critical review of Robin Hanson’s paper, “A Manipulator Can Aid Prediction Market Accuracy”.

We should not use prediction markets in the climate change problematic.

Chris F. Masse — Wed, 25 Nov 2009 21:17:07 +0000

Paul Hewitt:

While it may be possible to operate public policy prediction markets for some issues, their use in the climate change or global warming debate is questionable. Not only can there be no guarantee of manipulation-free markets, we wouldn’t even know if market predictions had been manipulated. If actual public policy were to depend on false readings from such markets, the potential for significant misallocation of resources is immense. It is simply too great a risk to consider at this time, in my opinion.