The idea that betting could help us gain clarity on some controversial scientific questions has first been proposed by George Mason economics professor Robin Hanson in 1992 in a paper entitled “Could Gambling Save Science” and available online here: http://hanson.gmu.edu/gamble.html

The benefits of creating prediction markets about controversial climate-change issues in particular is further developed on Nate Silver’s blog and in this presentation given at CalTech in 2004: http://us.newsfutures.com/home/environmentalFutures.html

Despite some limited evidence, the insistence on traders needing to know the direction of manipulation isn’t too compelling since the direction will be manifest insofar as the price is “wrong.” “Noise trader” is a politically loaded and misleading term. Misleading because it suggests that the mean effect will be zero, when in reality “noise trading” usually takes the form of feedback trading. Lack of feedback trading is a significant assumption in the Hanson manipulation papers. Fortunately, prediction markets have objective settlements at specified times, unlike traditional assets where the meaning of prices is open to interpretation, making them more prone to feedback trading and irrational booms and busts.

With prediction markets, conditions for manipulation are more favorable when the settlement is far off in time, and when there are subjective inputs to the settlement, e.g. in politics. A distant settlement simultaneously makes it less clear what the real price should be, and delays manipulator losses because there is less incentive to correct price. At the limit, a manipulator could introduce a price distortion when a contract is launched, only to reverse position for small liquidity-related loss immediately before settlement, thereby destroying the markets “integral” of error over time.

Another big assumption, also identified by Paul Hewitt, is that traders have equal account sizes. But maybe this isn’t a huge problem if settlement is forthcoming, and maybe the issue could be mitigated with additional exchange disclosures, such as the standard deviation of position sizes in a given market. While this could discourage liquidity as large traders would become paranoid about their positions, it is essentially a “soft” position limit, and traders would be forewarned of one-sided markets (which could of course be the result of someone well-informed, but I – the google-anonymous* writer – would bet that more concentration comes with more error on average… this can be tested by someone with the data, of course maintaining trader anonymity)

Even accounting for long-term settlement, feedback trading, semi-subjective settlements, and account size imbalances, it seems one would have to abide to an overly rigid tenet of “do no harm” to hold that prediction markets are, on net, a bad idea. (Do no harm is of course abhorrent to libertarians, and even doctors don’t actually follow such a rule.) Moreover, some pathologies like political self-fulfilling prophecy will only happen if prediction markets have already demonstrated their value and have become more popular. But even if one believes in their long term success, single pathologies can damage one’s reputation permanently… if one plans to die at a reasonable age.

[*Given the political climate, many firms have issued directives to employees to not engage in even the slightest appearance of impropriety, which might include blogging on manipulation.]

Crampton’s blog

While it may be possible to operate public policy prediction markets for some issues, their use in the climate change or global warming debate is questionable. Not only can there be no guarantee of manipulation-free markets, we wouldn’t even know if market predictions had been manipulated. If actual public policy were to depend on false readings from such markets, the potential for significant misallocation of resources is immense. It is simply too great a risk to consider at this time, in my opinion.

With all due respect, that testing was done on markets 30 days before they closed. As I noted, time had almost run out on each of the markets. Consequently, one might expect an almost perfect accuracy and calibration.

The problem with these long-term markets is that they are usually very inaccurate for most of their durations. It is only when the revelation is near that the market becomes accurate (and calibrated). Unfortunately, it is of little use to have an accurate prediction just before the outcome becomes known. In order to act upon the market’s prediction, we need to know that it is accurate sufficiently far in advance to do something about it. On this basis, these markets are not particularly useful.

I think we can all agree that each market is unique, in terms of the issue to be predicted, available information set, trader pool, etc… I don’t think you can take a sample of prediction markets from on marketplace (ideosphere), find they are calibrated, and conclude that *all* of the marketplace’s PMs are calibrated.

I don’t have the answers, here, but it would seem to me that you need to control for either the information set or the participant pool, to be able to make a general statement of calibration accuracy for any particular type of PM.

Paul Hewitt:

Just because there are long-term prediction markets does not mean that they are useful. Yes, there are a few long-term markets on ideosphere.com. Rather than dissect each one, I’ll look at two of them.

NUKE – started 10/10/1995 to predict whether nuclear weapons will be used before 1/1/2010. Ten years ago, it was still trading in the high 50s and 3-4 years ago in the 40s. Only in the last year (with time running out) is the market approaching zero (it’s at 2). This market will either be very accurate or wildly wrong. From a decision-maker’s viewpoint, this market is not very useful. If we assume the actual outcome will be NO, the market has only been accurate in the latest year. For most of the duration of this market, the prediction was very, very wrong. Had decisions been made based on earlier predictions, they would have been costly mistakes.

Terr10 – started 9/12/2001 to predict whether there would be another terrorist attach before 2010. Up to three years ago, the market was trading in the 60-70% range. Again, only in the last year has the prediction dropped to 3-5%. Had any government been relying on this market (until recently), they would have devoted far more resources to combating terrorism than were necessary.

Here we have two markets that are not particularly useful until the market is nearing its close. There’s no surprise here, time is running out in both markets. Interestingly, had the markets gone the other way – becoming almost certain of the event occurring, we should all have been concerned. But then again, there would have been little that could have been done to avert the disaster! Once again, these types of prediction markets prove they are not very useful for proper decision-making.

As for Robin Hanson’s statement that these ideosphere prediction markets are calibrated, except for a possible long shot bias, I have to ask… How do they prove this? You may be able to make that statement for certain types of markets that ideosphere runs, but you cannot generalize to all of their markets as if it were a brand attribute. At any rate, it appears highly unlikely that any of these long term markets is well-calibrated, because they are all very thinly traded. It is also questionable whether these markets are accurate, until very near the market closing, when the outcome will be revealed. Of course, by that time, the average guy on the street could give you that prediction.

No, I don’t think these long-term prediction markets are proven.

At ideosphere.com you will find eight claims that are over fifteen years old and still trading. One of them will be known in a few weeks. Calibration tests have been done on ideosphere prices showing moderate long shot bias I think.

ACT ONE: Nate Silver lunched with Robin Hanson, one month ago, to interview him for his future book about forecasting.

ACT TWO: Robin Hanson has published yet another post pumping up prediction markets for solving every problem on Earth —this time, yelling in bold that they are the solution to any global warming controversy.

ACT THREE: Nate Silver reads his post, and declares to the world that “Dr. Hanson” is the new Jesus.

MY TAKE: While it is true that prediction markets “incentivize accuracy”, they have been so far implemented only for short-term problematics (e.g., “Will Barack Obama get elected as US president?“). Plus, they have not made a difference —the non-business news media never cite the probabilities generated by the prediction markets.

I expected a less naive analysis from Nate Silver. He should read Midas Oracle before proclaiming to the world that long-term prediction markets are the solution: “The Robin Hanson-inspired, real-money, conditional prediction markets at InTrade are a world-wide disaster of thermonuclear proportion, concludes the long-time Robin Hanson fanboy who forked over real money to InTrade to have them set up and run.“

[...] I conclude those four contracts did not provide valuable information about how the [2008 US presidential] election would affect the world.

UPDATE: See the comments below, and Nate Silver’s second take.

Companies often ask me if prediction markets can forecast distant future topics. I tell them yes, [...]

-

Huh?

-

Here’s the second part of the statement:

[...] but that is not the place to test any doubts about prediction markets. To vet or validate prediction markets, you want topics where there will be many similar forecasts over a short time, with other mechanisms making forecasts that can be compared.

-