I am a “calibrationist”.

But I am strongly with Chris in this: Predicting the decision of a private committee is inherently a task ill-suited for prediction markets. Olympic Games, Nobel prizes, and similar events cannot be predicted. I will commend Chris for being stable in his position, for a long time. Chris, please keep the same tone when some market in the future will succeed in predicting the outcome. Even a broken clock is correct twice a day and eventually some market will hit the target for Nobel or the Olympic games.

If there is no public information to be aggregated, what are we trying to aggregate? Even if we run markets for 1000 years, I doubt that we will see any predictive power.

If we assume Ladbrokes is calibrated, given that Fema was running as the frontrunner with 2/1, in the next couple of years, the equivalent front runner “should” win to revert to the mean.

For Intrade, the 0-volume can be interpreted in two ways: (a) Pathetic participation (b) The traders knew that it is impossible to predict, and they played the lottery instead. [*]

[*] For years, InTrade have been created Nobel Prize prediction markets on the Thursday or Friday prior to the announcement on the next Monday. That’s crazy. InTrade should start off one or two months prior to the Nobel Prize announcements in October.

Addendum: See also the comments on this other thread.

Both the bookmakers and the prection markets are utterly useless in trying to divine who will get the Nobel prize of economics.

Below is the 2009 prediction post-mortem:

1. Bookmakers

Ladbrokes’s probabilities (odds) for the 2009 Nobel prize in economics:

Eugene Fama 2/1
Paul Romer 4/1
Ernst Fehr 6/1
Kenneth R. French 6/1
William Nordhaus 6/1
Robert Barro 7/1
Matthew J Rabin 8/1
Jean Tirole 9/1
Martin Weitzman 9/1
Chris Pissarides 10/1
Dale T Mortensen 10/1
Xavier Sala-i-Martin 10/1
Avinash Dixit 14/1
Jagdish N. Bhagwati 14/1
Robert Schiller [sic] 14/1
William Baumol 16/1
Martin S. Feldstein 20/1
Christopher Sims 25/1
Lars P. Hansen 25/1
Nancy Stokey 25/1
Peter A Diamond 25/1
Thomas J. Sargent 25/1
Dale Jorgenson 33/1
Paul Milgrom 33/1
Oliver Hart 40/1
Bengt R Holmstrom 50/1
Elhanan Helpman 50/1
Ellinor Ostrom 50/1
Gene M Grossman 50/1
Karl-Goran Maler 50/1
Oliver Williamson 50/1
Robert B Wilson 50/1

2. Betting Pools

Here is the betting in the Nobel pool at Harvard:

Robert Barro -10%
John Taylor – 8%
Paul Milgrom – 8%
Jean Tirole – 6%
Oliver Williamson – 6%
Martin Weitzman – 6%
Eugene Fama – 5%
Richard Thaler – 5%
Lars Hansen – 4%
Paul Romer – 4%

3. Prediction Markets

InTrade:

Previously: Nobel Prize for Economics 2009 Predictions

Paul Hewitt:

Hi Jed…

As for qualifications, I have been making business decisions for almost 30 years. I am a chartered accountant and a business owner. Starting in university and continuing to this day, I have been researching information needs for corporate decision making. As Chris points out, I’m not a salesperson for any of the software developers. In fact, if I have a bias, it is to be slightly in favour of prediction markets. That said, I still haven’t seen any convincing evidence that they work as promised by ANY of the vendors.

As for whether I have ever run or administered a prediction market, the answer is no. Does that mean I am not qualified to critique the cases that have been published? Hardly. You don’t have to run a PM to know that it is flawed. Those that do, end up trying to justify minuscule “improvements” in the accuracy of predictions. They also fail to consider the consistency of the predictions. Without this, EPMs will never catch on. Sorry, but that is just plain common sense.

The pilot cases that have been reported are pretty poor examples of prediction market successes. In almost every case, the participants were (at least mostly) the same ones that were involved with internal forecasting. The HP markets, yes, the Holy Grail of all prediction markets, merely showed that prediction markets are good at aggregating the information already aggregated by the company forecasters! They showed that PMs are only slightly better than other traditional methods – and mainly because of the bias reduction. Being slightly better is not good enough in the corporate world.

I think I bring a healthy skepticism to the assessment of prediction markets. I truly want to believe, but I need to be convinced. I am no evangelist, and there is no place for that in scientific research. Rather than condemn me for not administering a PM, why not address the real issues that arise from my analyses?

Paul Hewitt’s blog

Previously: The truth about CrowdClarity’s extraordinary predictive power (which impresses Jed Christiansen so much)

1. Carlos Graterol should publish what the politicians and editorialists were saying last week (the resignation calls were numerous);
2. The InTrade prediction market should have been created much, much earlier —when Glenn Beck started his ‘anti-czars’ guerrilla (at the beginning of August 2009).

Background

This paper presents the outcome of a study conducted at the Royal Devon and Exeter Hospital in which a prediction market was established in order to forecast demand for services. To the researcher’s knowledge, it does not appear that prediction markets have been previously utilized in a healthcare environment.

Purposes

The purpose of this study is to provide evidence for the effective use of prediction markets in a healthcare environment.

Methodology and Approach

The study was conducted over a period of one week, and involved sixty-five participants. Each was asked to provide an estimate for demand for services at the Royal Devon and Exeter Hospital. Characteristics gathered for each participant included level of education, occupation, directorate, number of years worked for the hospital, and number of years worked for the National Health Service.

Findings

The study confirms the effectiveness of prediction markets to forecast future events as overall hospital demand was forecasted with an error of only 0.3%. The prediction market was less successful in predicting demand for services for each department, which the researcher attributes to the small sample size and lack of diversity of participants. Additionally, only a very small percentage of the characteristics captured registered a statistically significant correlation with the accuracy of the estimate. Further studies should focus on different characteristics and/or use a larger sample size to either confirm or refute the existence of such characteristics.

Practical Implications

The findings of this work could potentially be used as an innovative way to augment the forecasting function for a wide range of healthcare facilities. With the preliminary success of this study to forecast demand, further research in the field is warranted.

Followed from:

- Crowdsourcing and Prediction Markets: Right-Timing Revisited

Via the Spigit guy

-

Gartner – Hype Cycle for Social Software, 2008 – (PDF file):

-

California Institute of Technology economist Charles Plott:

What you’re doing is collecting bits and pieces of information and aggregating it so we can watch it and understand what people know. People picked this up and called it the “wisdom of crowds” and other things, but a lot of that is just hype.

-

Previously:

Gartner: The “benefit” of enterprise prediction markets is “moderate” and “early users, who have begun to overestimate their accuracy and overall usefulness, are now somewhat disillusioned with the technology.”

[...] The hard part, the real hurdles to acceptance are (i) to get wide and consistent participation, (ii) to overcome the suspicion of decision makers who naturally don’t like to be second-guessed by subordinates, (iii) to get the organization to agree to ask important questions instead of trivial ones.

Go also read Paul Hewitt’s take on Emile’s latest paper.

Prediction Markets

Analysis By: Matthew Cain; Nikos Drakos

Definition: Prediction markets are betting mechanisms established to forecast the outcome of an ongoing issue of contention, such as quarterly sales or product delivery dates. They can also be used to prioritize ideas (users bet on which ideas have the greatest chance of success) or to assess risk (users bet on which paths carry the least risk). While popular on the Web for predicting election results (among other things), and popular among gamblers (will an EU nation boycott the Olympics, for example), [prediction] markets are now entering the business world. Users are typically given a finite amount of money (often play money) and they buy shares in the outcome they believe most likely. Outcomes that attract the most money create a forecast based on a consensus opinion which, in many cases, has proven more accurate than traditional means of forecasting.

Position and Adoption Speed Justification: While the value of [prediction] markets are well codified (used in the proper circumstances), mainstream awareness of the technology is minimal. Larger organizations, mostly technology and pharmaceutical firms, were the first to adopt the technology, but it has been slow to spread to other vertical markets. [Prediction] markets are positioned on the upper part of the Trough of Disillusionment because early users, who have begun to overestimate their accuracy and overall usefulness, are now somewhat disillusioned with the technology.

User Advice: Organizations looking to improve forecasting of contentious issues where multiple users have valid insight into the issue are likely to benefit from investments in [prediction] markets. A learning curve will be in effect as organizations determine which circumstances and participants are most likely to result in accurate forecasts. Organizations will also strive to find the most appropriate incentives to yield the most accurate forecasts: one theory, for example, holds that use of fake money results in inferior results compared to real money. Besides numerous premises-based commercial (and open-source) vendors, there are multiple suppliers that offer [web-]hosted [prediction] markets, enabling exposure to the technology for minimal investment.

Business Impact: [Prediction] markets can help more-accurately model outcomes for sales efforts, product delivery dates, product uptake rates, manufacturing capacity needs, product ideas, marketing campaigns and competitive actions, for example. It is unlikely to be of help when the number of knowledgeable [bettors] is limited, when outcomes have a high degree of predictability, or when the outcome can arbitrarily be determined by a small number of people. Prediction markets are most valuable when they are used to rationalize a large number of disparate viewpoints. Potential users should start with pilot programs so they can compare the results with traditional forecasting mechanisms.

Benefit Rating: Moderate

Market Penetration: Less than 1% of target audience

Maturity: Emerging

Sample Vendors: [See the updated Midas Oracle list]

IDEAS: Any individual expert is likely to be wrong. What happens when you put a bunch of them in the same room? Does the consensus come up with a better prediction than the individual?

TETLOCK: Usually yes. Jim Surowiecki wrote that very clever book on “The Wisdom of Crowds.” . . . Sometimes social scientists call it the miracle of aggregation. It’s an interesting fact that the average of the experts’ predictions often outperforms the individuals from whom the average was derived.

IDEAS: But the consensus can be wrong, too, as in the case of the widely shared view that Iraq had weapons of mass destruction. How does that happen?

TETLOCK: The crowd can certainly be wrong. Sometimes, the crowd can be dramatically wrong. . . . One big danger is if you’re dealing with an ideologically, intellectually homogeneous group, and they’re all talking to each other. They can really talk each other into some quite extreme positions.

-

Philip Tetlock interviewed (this year):

What makes some forecasters better than others?

The most important factor was not how much education or experience the experts had but how they thought. You know the famous line that [philosopher] Isaiah Berlin borrowed from a Greek poet, “The fox knows many things, but the hedgehog knows one big thing”? The better forecasters were like Berlin’s foxes: self-critical, eclectic thinkers who were willing to update their beliefs when faced with contrary evidence, were doubtful of grand schemes and were rather modest about their predictive ability. The less successful forecasters were like hedgehogs: They tended to have one big, beautiful idea that they loved to stretch, sometimes to the breaking point. They tended to be articulate and very persuasive as to why their idea explained everything. The media often love hedgehogs.

How do you know whether a talking head is a fox or a hedgehog?

Count how often they press the brakes on trains of thought. Foxes often qualify their arguments with “however” and “perhaps,” while hedgehogs build up momentum with “moreover” and “all the more so.” Foxes are not as entertaining as hedgehogs. But enduring a little tedium is worth it if you want realistic odds on possible futures.

So if you were looking for a money manager, you’d want a fox?

If you want good, stable long-term performance, you’re better off with the fox. If you’re up for a real roller-coaster ride, which might make you fabulously wealthy or leave you broke, go hedgehog.

But it was doomster hedgehogs like money managers Robert Rodriguez and Jeremy Grantham who first saw the crisis coming.

Hedgehogs are sometimes way, way out front. But they can also be way, way off.

-

His book: Expert Political Judgment: How Good Is It? How Can We Know?