Paul Hewitt sends Robin Hanson packing.
Just because there are long-term prediction markets does not mean that they are useful. Yes, there are a few long-term markets on ideosphere.com. Rather than dissect each one, I’ll look at two of them.
NUKE – started 10/10/1995 to predict whether nuclear weapons will be used before 1/1/2010. Ten years ago, it was still trading in the high 50s and 3-4 years ago in the 40s. Only in the last year (with time running out) is the market approaching zero (it’s at 2). This market will either be very accurate or wildly wrong. From a decision-maker’s viewpoint, this market is not very useful. If we assume the actual outcome will be NO, the market has only been accurate in the latest year. For most of the duration of this market, the prediction was very, very wrong. Had decisions been made based on earlier predictions, they would have been costly mistakes.
Terr10 – started 9/12/2001 to predict whether there would be another terrorist attach before 2010. Up to three years ago, the market was trading in the 60-70% range. Again, only in the last year has the prediction dropped to 3-5%. Had any government been relying on this market (until recently), they would have devoted far more resources to combating terrorism than were necessary.
Here we have two markets that are not particularly useful until the market is nearing its close. There’s no surprise here, time is running out in both markets. Interestingly, had the markets gone the other way – becoming almost certain of the event occurring, we should all have been concerned. But then again, there would have been little that could have been done to avert the disaster! Once again, these types of prediction markets prove they are not very useful for proper decision-making.
As for Robin Hanson’s statement that these ideosphere prediction markets are calibrated, except for a possible long shot bias, I have to ask… How do they prove this? You may be able to make that statement for certain types of markets that ideosphere runs, but you cannot generalize to all of their markets as if it were a brand attribute. At any rate, it appears highly unlikely that any of these long term markets is well-calibrated, because they are all very thinly traded. It is also questionable whether these markets are accurate, until very near the market closing, when the outcome will be revealed. Of course, by that time, the average guy on the street could give you that prediction.
No, I don’t think these long-term prediction markets are proven.
Hewitt is confusing accuracy with calibration. I’ve tested the calibration of ideosphere markets as a whole (short and long term together) and they are indeed calibrated; see:
http://artificialmarkets.com/am/pennock-2001-science/
That’s a good idea to test the long-term markets separately.
QUOTE
David Pennock’s anaysis of ideosphere.com long-term prediction markets shows they were calibrated 30 days before the outcome was revealed. In no way does this imply that the markets were calibrated at earlier times. It is possible that they were, but we may never be able to determine this. And, even if they are calibrated far in advance, the flat distributions don’t provide useful information for decision-making, they tell us we need more information or the outcome is too uncertain to predict. Either way, it does not support the potential of futarchy.
UNQUOTE
http://www.overcomingbias.com/2010/01/hewitt-on-futarchy.html#comment-440064
Thanks.
With all due respect, that testing was done on markets 30 days before they closed. As I noted, time had almost run out on each of the markets. Consequently, one might expect an almost perfect accuracy and calibration.
The problem with these long-term markets is that they are usually very inaccurate for most of their durations. It is only when the revelation is near that the market becomes accurate (and calibrated). Unfortunately, it is of little use to have an accurate prediction just before the outcome becomes known. In order to act upon the market’s prediction, we need to know that it is accurate sufficiently far in advance to do something about it. On this basis, these markets are not particularly useful.
I think we can all agree that each market is unique, in terms of the issue to be predicted, available information set, trader pool, etc… I don’t think you can take a sample of prediction markets from on marketplace (ideosphere), find they are calibrated, and conclude that *all* of the marketplace’s PMs are calibrated.
I don’t have the answers, here, but it would seem to me that you need to control for either the information set or the participant pool, to be able to make a general statement of calibration accuracy for any particular type of PM.