Nate Silver becomes a little Robin Hanson fanboy… after just a lunch with the Master.
ACT ONE: Nate Silver lunched with Robin Hanson, one month ago, to interview him for his future book about forecasting.
ACT TWO: Robin Hanson has published yet another post pumping up prediction markets for solving every problem on Earth —this time, yelling in bold that they are the solution to any global warming controversy.
ACT THREE: Nate Silver reads his post, and declares to the world that “Dr. Hanson” is the new Jesus.
MY TAKE: While it is true that prediction markets “incentivize accuracy”, they have been so far implemented only for short-term problematics (e.g., “Will Barack Obama get elected as US president?“). Plus, they have not made a difference —the non-business news media never cite the probabilities generated by the prediction markets.
I expected a less naive analysis from Nate Silver. He should read Midas Oracle before proclaiming to the world that long-term prediction markets are the solution: “The Robin Hanson-inspired, real-money, conditional prediction markets at InTrade are a world-wide disaster of thermonuclear proportion, concludes the long-time Robin Hanson fanboy who forked over real money to InTrade to have them set up and run.“
UPDATE: See the comments below, and Nate Silver’s second take.
Where to begin?!
Yes, prediction markets “incentivize accuracy”, but there is no guarantee of success. So far, we know that they *can* work in short (actually very short) term issues. There is absolutely no evidence that they work in any long (let alone very long) term issues. Worse, no one is looking into this issue.
As it stands, PMs cannot predict even one year hence. How in the world (warming or cooling) does anyone expect them to predict 10, 50, 100 or 1,000 years into the future?
The PM incentive for accuracy breaks down the longer the period to the payoff. Plain old human nature at play. People will not wait long periods for gratification, if there are shorter options available (other PMs, casinos, etc…). It is not a coincidence that craps is the fastest game in the casino.
Given the relatively small amounts that can be bet on real money PMs, on very important issues (global warming), it is easy to find other incentives that would influence the accuracy of the markets.
It would depend on the intended use of the PM results, but you could see large carbon emitters exerting downward price pressure in a market that attemts to predict the likelihood of any signficant harm being caused (which would lessen the impact of government intervention, saving the emitter much larger compliance costs). The point, here, is that the “incentivization” only works if there are no, even larger, incentives to be inaccurate.
Non-business media never cite PM predictions, because PMs are not proven to work, yet. They must be reliable – both accurate and consistent. Neither has been proven at this point.
Robin Hanson’s conditional prediction markets are promising, but they aren’t going to get off the ground until, at the very least, prediction markets are proven to be consistently accurate. I believe that it can be done for a narrow range of topics, but not for a wide range of social or political issues. Until the markets are consistently accurate, they must be considered “broken tools” (some work, sometimes). Making the markets more complex (conditional PMs) will not “repair” the broken tools. This is a prescription for bruised thumbs, or should I say egos.
At ideosphere.com you will find eight claims that are over fifteen years old and still trading. One of them will be known in a few weeks. Calibration tests have been done on ideosphere prices showing moderate long shot bias I think.
Just because there are long term prediction markets does not mean that they are useful. Yes, there are a few long term markets on ideosphere.com. Rather than dissect each one, I’ll look at two of them.
NUKE – started 10/10/1995 to predict whether nuclear weapons will be used before 1/1/2010. Ten years ago, it was still trading in the high 50s and 3-4 years ago in the 40s. Only in the last year (with time running out) is the market approaching zero (it’s at 2). This market will either be very accurate or wildly wrong. From a decision-maker’s viewpoint, this market is not very useful. If we assume the actual outcome will be NO, the market has only been accurate in the latest year. For most of the duration of this market, the prediction was very, very wrong. Had decisions been made based on earlier predictions, they would have been costly mistakes.
Terr10 – started 9/12/2001 to predict whether there would be another terrorist attach before 2010. Up to three years ago, the market was trading in the 60-70% range. Again, only in the last year has the prediction dropped to 3-5%. Had any government been relying on this market (until recently), they would have devoted far more resources to combating terrorism than were necessary.
Here we have two markets that are not particularly useful until the market is nearing its close. There’s no surprise here, time is running out in both markets. Interestingly, had the markets gone the other way – becoming almost certain of the event occurring, we should all have been concerned. But then again, there would have been little that could have been done to avert the disaster! Once again, these types of prediction markets prove they are not very useful for proper decision-making.
As for Robin Hanson’s statement that these ideosphere prediction markets are calibrated, except for a possible long shot bias, I have to ask… How do they prove this? You may be able to make that statement for certain types of markets that ideosphere runs, but you cannot generalize to all of their markets as if it were a brand attribute. At any rate, it appears highly unlikely that any of these long term markets is well-calibrated, because they are all very thinly traded. It is also questionable whether these markets are accurate, until very near the market closing, when the outcome will be revealed. Of course, by that time, the average guy on the street could give you that prediction.
No, I don’t think these long term prediction markets are proven.
Can you show that some other source is *more* accurate than this source? Your approach of just looking at the final price and then complaining why didn’t they see that earlier seems very susceptible to hindsight bias. Please also consider we have long term bond futures whose prices make long term forecasts.
Robin, just because we don’t have another method of accurately predicting an outcome doesn’t mean we have to be so appreciative when a prediction market comes up with a forecast just before the outcome is revealed. My point is that it isn’t very useful to know something just before it happens (usually).
It is only useful, if, knowing the outcome in advance, I am able to change my course of action, resulting in some net benefit. If I am unable to take advantage of this information, it becomes a trivial pursuit. Nice to know, but… who cares?
Paul Hewitt’s (critical) review of Robin Hanson’s paper, “A Manipulator Can Aid Prediction Market Accuracy“.
http://torontopm.wordpress.com/2009/11/26/is-it-enough-to-provide-incentives/
Nate Silver’s second take:
http://www.fivethirtyeight.com/2009/11/case-for-climate-futures-markets-ctd.html
“If I am unable to take advantage of this information, it becomes a trivial pursuit. Nice to know, but… who cares?”
As I said many times, *only* the business media (and generally speaking, the free-market fanboys) are *sometimes* citing the probabilities generated by the prediction markets.