Both the bookmakers and the prection markets are utterly useless in trying to divine who will get the Nobel prize of economics.
Below is the 2009 prediction post-mortem:
1. Bookmakers
Ladbrokes’s probabilities (odds) for the 2009 Nobel prize in economics:
Eugene Fama 2/1
Paul Romer 4/1
Ernst Fehr 6/1
Kenneth R. French 6/1
William Nordhaus 6/1
Robert Barro 7/1
Matthew J Rabin 8/1
Jean Tirole 9/1
Martin Weitzman 9/1
Chris Pissarides 10/1
Dale T Mortensen 10/1
Xavier Sala-i-Martin 10/1
Avinash Dixit 14/1
Jagdish N. Bhagwati 14/1
Robert Schiller [sic] 14/1
William Baumol 16/1
Martin S. Feldstein 20/1
Christopher Sims 25/1
Lars P. Hansen 25/1
Nancy Stokey 25/1
Peter A Diamond 25/1
Thomas J. Sargent 25/1
Dale Jorgenson 33/1
Paul Milgrom 33/1
Oliver Hart 40/1
Bengt R Holmstrom 50/1
Elhanan Helpman 50/1
Ellinor Ostrom 50/1
Gene M Grossman 50/1
Karl-Goran Maler 50/1
Oliver Williamson 50/1
Robert B Wilson 50/1
2. Betting Pools
Here is the betting in the Nobel pool at Harvard:
Robert Barro -10%
John Taylor – 8%
Paul Milgrom – 8%
Jean Tirole – 6%
Oliver Williamson – 6%
Martin Weitzman – 6%
Eugene Fama – 5%
Richard Thaler – 5%
Lars Hansen – 4%
Paul Romer – 4%
3. Prediction Markets
Previously: Nobel Prize for Economics 2009 Predictions
Fascinating. Ladbrokes spells her name incorrectly! Harvard and intrade don’t even list her as a potential winner. Just shows you that there is a lot more to prediction markets than building the mechanism. These were the latest in pathetic examples of prediction markets.
Interestingly, Elinor Ostrom won the prize for her studies in collective decision-making by the users of public properties. She found that a group of public property users had better information than bureaucrats and were able to make better decisions regarding the management of the resources.
All of this shows that collective decision-making *can* work, it’s just that prediction markets have yet to show that they *do* work.
I can hardly wait for the prediction market calibration proponents to claim that these were not market failures. On the contrary, these outcomes show, precisely, that prediction markets do, in fact, work.
Of course, they may be right. I doubt that I will live another 50 years or so to test the calibration accuracy of similar prediction markets. It is rather interesting that not one, but two, 50 – 1 longshots claimed the prize. There is no calibration accuracy in these types of markets, and there is no consistency across a variety of similar prediction markets (and there should be, if these types of prediction markets really do work).
What do you make of this:
http://blog.inklingmarkets.com/2009/10/catching-crazies.html
I agree with Adam Siegel when he writes:
Sounds like Adam is a “calibrationist” (new term), yet he seeems to understand that markets cannot “answer” all questions. When we try to make them provide an answer to the unknowable outcome, we open ourselves up to the inevitable letdown.
With Adam’s stature in the prediction market “industry”, can’t he help steer research in the right direction?
And, where’s Robin Hanson with all of this going on? Is he in Stockholm trying to sell the Nobel selection committee on his decision-making software solution? I wonder.
I am a “calibrationist”.
But I am strongly with Chris in this: Predicting the decision of a private committee is inherently a task ill-suited for prediction markets. Olympic Games, Nobel prizes, and similar events cannot be predicted. I will commend Chris for being stable in his position, for a long time. Chris, please keep the same tone when some market in the future will succeed in predicting the outcome. Even a broken clock is correct twice a day and eventually some market will hit the target for Nobel or the Olympic games.
If there is no public information to be aggregated, what are we trying to aggregate? Even if we run markets for 1000 years, I doubt that we will see any predictive power.
If we assume Ladbrokes is calibrated, given that Fema was running as the frontrunner with 2/1, in the next couple of years, the equivalent front runner “should” win to revert to the mean.
For Intrade, the 0-volume can be interpreted in two ways: (a) Pathetic participation (b) The traders knew that it is impossible to predict, and they played the lottery instead.
By the way, how do you like Felix Salmon’s take:
The non-ironic Nobel
http://blogs.reuters.com/felix-salmon/2009/10/12/the-non-ironic-nobel/
The biggest thing to be learned from these markets is that there is a spectrum of elements that will likely make markets more (or less) accurate.
In general, more members/voters will be better than fewer (IOC markets better than Nobel markets). More objective criteria will be better than less (IOC markets better than Nobel markets). Constrained choices will be better than unconstrained choices (choice of 4 cities versus choice of 100’s of economists/chemists/etc). Voters signaling choices before a vote is better than if they don’t (US Congressional votes versus either IOC/Nobel markets). These elements not only affect relative accuracy, but also the risk and thus liquidity in a marketplace. (Panos’s option [b] above)
Yes, decisions made by “secretive committees” are poor choices for prediction markets. But there are some very secretive committees & choices (aka, Nobel Peace Prize committee) and less secretive committees & choices (IOC membership). While neither will likely be as accurate as traditional open prediction markets, there is a spectrum of accuracy involved. The IOC markets will be on the higher end of that spectrum (because of a large membership, limited options, specified criteria, known biases, etc.), and the Nobel markets will be on the lower end of that spectrum.
Jed, this makes sense.
Potentially you can add Academy Awards (Oscars) as a even more open committee. I am wondering what is the proper modeling approach to capture this phenomenon in a more principled manner. Are you aware of any work along these lines?
Hi, Panos.
I definitely agree with your Academy Awards as another good example. Unfortunately, I’m not aware of any work along these lines. I got started thinking about all this to try and distinguish why some markets would be worse than others, and came up with some of the ideas above.
It would be great to see some of this modeled, but I’m not sure how to go about it with existing data, as there is so little of it in many cases (such as one IOC decision every two years). A good first step would be to develop a comprehensive list of factors that could affect accuracy.
“Academy Awards (Oscars) as a even more open committee”
Last year, everybody predicted Mickey Rourke, and he didn’t get it, coz Hollywood saw it as a jerk. Hollywood does not see things as Main Street does. (See the Polanski affair, for instance.)
So trying to divine Hollywood’s future decisions is tricky.
Another example of things difficult to divine: Selection of a VP in US presidential election.
“So trying to divine Hollywood’s future decisions is tricky.”
Tricky, but not impossible. HSX has an excellent record with both Oscar winners as well as the nominations (which are more difficult because there is no limit on possibilities).
“Another example of things difficult to divine: Selection of a VP in US presidential election.”
Very true. In my mind that would be generally equivalent to a Nobel Peace Prize winner. (Very small committee choosing from a very wide pool of candidates in near-total secrecy.)
But again, there is a spectrum of accuracy in running markets on these decisions.
Jed’s made a few good observations. However, one doesn’t make any sense. In particular, his comment about the size of the member/voter group influencing the accuracy of the prediction market is hard to follow. The member/voter group determines the actual outcome. A prediction market attempts to predict that same outcome. The two groups are entirely independent of one another.
Then, he mentions that secretive (or not) committees will not be as accurate as traditional open prediction markets. Perhaps I’m misreading this, but it makes no sense to compare these two methods.
Panos asked whether there might be a more principled method of determining the accuracy of prediction markets. I’ve written a bit about this. All that needs to be done is to fill in the details.
I’ve posted more detailed comments on my blog:
http://torontopm.wordpress.com/2009/10/20/more-public-prediction-market-failures/
Paul, I think Jed’s comment is about whether we can predict the *accuracy of the PM*, when a PM is trying to predict the outcome of a committee decision. So, it is kind of a “meta” question.
To align our vocabularies: With accuracy, in this setting, I think that Jed (and myself) mean the degree of calibration. So, for “open” committees where information flows, the degree of calibration will be high. For highly-secretive committees, the degree of calibration will be low.
For a perfectly calibrated market, the scatterplot of price-vs-”fraction-of-times-outcome-occurs” is a perfect diagonal. For a market that generates random decisions, the scatterplot of price-vs-”fraction-of-times-outcome-occurs” is an almost set of points in the rectangle.
The statement about “secretive (or not) committees” refers to how accurately we can predict their outcome, using prediction markets. Not how well a committee can predict its own decision.
In other words, the question is: Given the characteristics of the event, can we predict the degree of calibration of the corresponding prediction market?
You correctly mention that it depends on the information completeness. But can we operationalize and quantify the notion of information completeness? Can we quantify how information completeness affects the degree of calibration of the corresponding prediction markets?
Panos, I think I stand corrected, regarding Jed’s comments. Though, it is rather curious that we would be trying to predict the accuracy of the prediction from a market, don’t you think?
I don’t know whether it will be possible to quantify information completeness. I suspect that we will not be able to do so. However, I do think, logically, that when information completeness is substantial, the market outcomes will be fairly highly correlated. I still don’t think this will mean anything, where the market is comprised of discrete options (as discussed previously).
In enterprise prediction markets, it should be possible to estimate the level of calibration for similar markets, over time. I doubt we will be able to generalize the findings to other types of prediction markets, but we should be able to develop calibration indices for each type of market.
“Though, it is rather curious that we would be trying to predict the accuracy of the prediction from a market, don’t you think?”
On a first sight, yes, it seems more like a capricious academic game.
However, being able to tell “how well-calibrated a PM on event X will be, given the characteristics of the event X” is an important question to answer. It explains whether it makes sense to deploy a market.
Qualitatively, I have a feeling on how to approach this (open committee vs closed committee, secrecy etc). It is even possible to empirically estimate the “degree of calibration”, for every event-type, using past markets. Then we can run, say, regressions to see how the degree of calibration is affected by various characteristics of the event.
But I have no clue how to even attempt to model this theoretically, as this requires to be able to somehow quantify information completeness.
“It is even possible to empirically estimate the “degree of calibration”, for every event-type, using past markets. Then we can run, say, regressions to see how the degree of calibration is affected by various characteristics of the event.
But I have no clue how to even attempt to model this theoretically, as this requires to be able to somehow quantify information completeness.”
I agree that we will need to be able to estimate the accuracy of a prediction market before it is run ‘live’. However, I do not think we will be able to analyse “similar” types of markets to do so. More likely, we will have to determine the calibration for virtually identical prediction markets. For example, an EPM to forecast quarterly sales of a corporate division. The same EPM would be run every quarter and the participants would be very similar in each one.
Finally, I agree with the information completeness issue. Presently, other than using calibration accuracy as evidence of information completeness, I’m not sure how we might quantify this very subjective attribute. Few, if any, seem to be working on this issue.
I’m not sure this will be helpful, but I found it very interesting that, in the week before the Palin nomination, the probability of ANY woman being the nominee went from the teens to the 30s. The markets basically missed Palin, but may have had a sense of what was going on. So when we talk about completeness, there is actual inside information, and there’s a rational assessment of the attributes — with VP nominations, often the desire for a balanced ticket in terms of gender, race, geography, areas of expertise, etc.
If there were actual contracts for these predicates or at least an interface for their implied prices, that might have an outside shot of helping with some of these questions.