Come to the wonderful world of collective intelligence, wisdom of crowds, and prediction markets!… The sun shines bright, the market-generated predictions are vastly superior to the polls as election predictors, and the track record of the public prediction markets stretches as far as the eye can see. There are opportunities aplenty in the field of prediction markets, and the trading technology is cheap. Every working enterprise can have its own internal prediction exchange, and inside every exchange, a set of enterprise prediction markets that correctly predicts the future of business, which their happy, all-American CEO listens to. Life is good in the magic world of prediction markets… it’s paradise on Earth.
Ha! ha! ha! ha!… That’s what they tell you, anyway… —because they are selling an image (just as Bernie Madoff did). They are selling it thru their vendor websites, vendor conferences, vendor-inspired articles in blogs, newspapers and magazines, and interviews of vendor data-fed professors in the media.
The prediction market technology is not a disruptive technology, and the social utility of the prediction markets is marginal. Number one, the aggregated information has value only for the totally uninformed people (a group that comprises those who overly obsess with prediction markets and have a narrow cultural universe). Number two, the added accuracy (if any) is minute, and, anyway, doesn’t fill up the gap between expectations and omniscience (which is how people judge forecasters). In our view, the social utility of the prediction markets lays in efficiency, not in accuracy. In complicated situations, the prediction markets integrate expectations (informed by facts and expertise) much faster than the mass media do. Their accuracy/efficiency is their uniqueness. It is their velocity that we should put to work.
-
Here’s now our definition of prediction markets:
-
A prediction market is a market for a contract that yields payments based on the outcome of a partially uncertain future event, such as an election. A contract pays $100 only if candidate X wins the election, and $0 otherwise. When the market price of an X contract is $60, the prediction market believes that candidate X has a 60% chance of winning the election. The price of this event derivative represents the imputed perceived likelihood of the partially uncertain future outcome (i.e., its aggregated expected probability). A 60% probability means that, in a series of events each with a 60% probability, the favored outcome is expected to occur 60 times out of 100, and the unfavored outcome is expected to occur 40 times out of 100.
Each prediction exchange organizes its own set of real-money and/or play-money markets, using either a CDA or a MSR mechanism —with or without an automated market maker.
Prediction markets enable us to attain collective intelligence. Prediction markets produce dynamic, objective probabilistic predictions on the outcomes of future events by aggregating disparate pieces of information that the traders bring when they agree on prices. The event derivative traders are informed by the primary indicators (i.e., the primary sources of information), like the polls, for instance. These informed speculators then execute their transactions based on their anticipations about the future —anticipations that will be either confirmed or infirmed.
The value of a set of prediction markets consists in the added accuracy that these prediction markets provide relative to the other meta predictive mechanisms, times the value of accuracy in improved decisions, minus the cost of maintaining these prediction markets, relative to the cost of the other meta predictive mechanisms. A highly accurate set of prediction markets has little value if some other meta predictive mechanism(s) can provide similar accuracy at a lower cost, or if very few substantial decisions are influenced by accurate predictions on its topic.
-
PS: I am updating a bit the content of this webpage, over time —so as to finesse the message.
In the world of business it is all about accuracy. You should know this.
I’ve been working on a project for almost 4 years now, together with 200 other people. It should have been completed 2 years ago. That’s a lot of tax money!
-
Of course, most people already knew about this, because of the underlying structural problems we were facing. The management however, didn’t have a clue. And it seems they don’t want to know, they do not listen. They are the true believers of the modern world, and hey, you don’t get to become a hot shot manager if you are not a believer. Failure is not in their dictionary, reality less so. But that doesn’t matter, simply tell people what they want to hear, and you will make it. At least in the short term, and who cares about the long term. In fact, who cares about wasting tax payers’ dollars anyway?
-
You look at it from your perspective, Chris, as a reporter. As a reporter, it is all about speed. But the art is to include all sides of the truth, and when done by one individual, can result in magic.
-
This is becoming a long-winded post. Other crucial aspects of prediction markets are cost-effectiveness and indeed… social utility! All depending on your point of view.
Markets will be of more value where there isn’t already some underlying quantified dataset, as there is with elections (polls) and disease (hospital records). In those cases, a lot of their relative value lies in quick integration of facts, but speed without accuracy isn’t desirable. Also in market-types that repeat over time, marginal advantages in accuracy become important. Consider that assumptions about transaction costs can be crucial to the profitability of a trading strategy.
The questions you ask are healthy, but I wouldn’t sell prediction markets short now.. the time to do that was a year ago.
Medemi,
“In the world of business it is all about accuracy.”
The prediction markets are very minimally more accurate than the other methods.
“quick integration of facts, but speed without accuracy isn’t desirable”
-
You have “speed without accuracy”, sometimes, in the early life of some mid-term prediction markets. Like the 2008 Dem primary prediction market, where we saw the Clinton bubble —which some interpreted as manipulation.
-
From September 2008 to Election Day, I don’t think you could say that there was “speed without accuracy” on the 2009 presidential prediction market.
Chris,
“The prediction markets are very minimally more accurate than the other methods.”
http://blog.inklingmarkets.com/2008/02/benefits-of-prediction-markets-in-it.html
You have NO idea of the extent of politics involved in IT project management when multiple companies (from different countries) are working “together”.
Hi, Chris. In many ways I agree with you. Prediction markets accuracy has in some cases been oversold. They are a tool, not a panacea.
-
But sometimes, even if prediction markets only provide a marginally more accurate forecast, the problem they’re addressing is so important that it’s a great return on investment. Other times, prediction markets might provide a highly accurate forecast, but of an issue that doesn’t really matter. Each company needs to think about how important accuracy is for each problem they’re looking to solve.
-
But I agree with Medemi when he talks about the problems with politics in an organisation. There are times where a prediction market could cut through these issues.
-
Finally, there’s an interesting opportunity for companies to gain really valuable insights not from the market results, but from the data behind the results. Bo and Google have already published some data in this vein, and from what I understand are working on it in further detail.
-
In summary, while I agree that marketing prediction markets tends to ignore the complexity of the solution, they are still valuable tools when used in context.
I called the death of these markets eighteen months ago. Do I get a prize for my predictive ability!
>> “But I agree with Medemi when he talks about the problems with politics in an organisation. There are times where a prediction market could cut through these issues.”
-
Designs by one of our suppliers are updated twice a week now. Previous deadlines were way off, so now we work on a trial-and-error basis. Politics dictate we pick it up and make the best of it, but it will still be rubbish. Sentiment is dire.
If I make you feel depressed a bit you now understand why.
-
Not sure what prediction markets could do, but I would say it would have given us better estimates and more realistic predictions.
-
Division of responsibilty is a key factor in overcoming politics and related issues. But you would still have to convince all related parties of the virtues of prediction markets, and ask commitment from them. A hell of a job, since people (generally) tend to believe they know better.
Jed,
Accuracy and efficiency? Where do prediction markets can show something for it? That’s the question.
Niall, I am calling the death of the accuracy argument —I am saying investigating efficiency will be more useful.
-
Let’s reboot the prediction markets. They are not dead. They should be rejuvenated.
-
Medemi,
One example about PM’s cutting through politics is the story that Todd Proebsting told at the Yahoo confab a while back. Essentially the project was way behind, but internal politics prevented that from being out in the open. A prediction market on the outcome brought a number of issues to light.
-
Now in that case a prediction market didn’t resolve the issues, and I have no idea how accurate the final result was. But it brought to light a significant issue in the project.
-
Chris, I’m not sure I understand your question. A number of empirical studies have shown that PM’s can be the most accurate forecasting method, though not always and the amount varies depending on the application. I agree that accuracy matters, but only in context. (How much more accurate, how valuable is the information, etc.)
>> “A prediction market on the outcome brought a number of issues to light.”
-
Yup, I forgot to mention that. That’s probably one of the most important aspects.
Heroism will get your ass fired, so most people keep quiet. They should be able to vote.
“A number of empirical studies have shown that PM’s can be the most accurate forecasting method”
By a cat’s whisker. (+2.7%, versus the polls)
And that’s been debated:
http://www.stat.columbia.edu/~cook/movabletype/archives/2007/12/are_political_m.html
-
So, you’re still an old-school man, I see.
-
The key value in any type of prediction is accuracy – how can an inaccurate prediction, even delivered early be of value?
The value of the accuracy of a prediction solution is in relation to 2 factors: the value of nailing this particular metric (how much would you gain by having perfect knowledge?) and the accuracy and cost of the next best solution.
Taking political polls, the next best solution is a well designed, well researched, well funded, large scale, professionally managed survey (Gallup etc). That our simple prediction market tools can come anywhere close on accuracy is a huge credit.
Regarding predictions within the corporation. Absolute knowledge is of incredible value, and often the next best solution is an guesstimate by the manager responsible for the project or business unit (who is not rewarded on accuracy, rather than on reputation management and achievement vs expectations – see Medimi’s example above). In myriad examples – Intel, HP, EA etc, a well designed prediction market has delivered significantly more accurate forecasts.
Prediction markets are not dead……………….
Ah, you’re thinking exclusively of election markets. Not every type of market has a data source like polls to compare results to… I’ve worked on a number of markets where the PM is the first time there’s even been an attempt to forecast! If you look at a broader range PM’s are still a very useful tool.
-
That, and like I said above, small differences can often matter a great deal. If you’re trying to forecast sales of a product with yearly revenues of $100 million (that has supply-chain costs like warehousing, etc.) then a 2.5% increase in forecasting quality can save a company $1 million a year. This is based off a paper written by an expert in the forecasting field (who has likely not done any work in prediction markets).
-
Medemi,
The best part about prediction markets for this is that while the result is public, the trades themselves aren’t. (Though can be inferred, of course.) We’re certainly on the same page, here.
Agree with Jed immediately above.
However, to add fuel to the over-with crowd, see NYT on how Google decides to kill products. No mention of PMs.
Jed, the PMs are not the only tool possible in the situation you describe. One could average the expectations of operators/agents/experts/employees, and get the same result that PMs got.
Chris, I absolutely agree. When deciding to use PM’s, good managers should think about the costs involved compared to the benefits possible. In some cases PM’s will come out on top, in many cases they won’t.
-
On this note, I love the story (I believe it’s true) of how Robin Hanson originally ran PM’s… just orders on a blackboard left for everyone to see. Just as effective as any computerised system, depending on the company/market.
Prediction markets: all about velocity, not accuracy?
http://www.stat.columbia.edu/~cook/movabletype/archives/2009/02/prediction-mark-1.html
-
Chris Masse: The truth about prediction markets
http://knowledgeproblem.com/2009/02/14/chris-masse-the-truth-about-prediction-markets/
Mike,
About Google, I heard the rumor that the man formerly in charge of their EPMs is no longer in charge of them… I have no idea whether it is true. Something we will have to check.
Jed,
-
Obviously, there has been a sudden drop of interest from the part of the media for the prediction markets since November 2008. For the good reason that the political prediction markets did not do better than the polls. All the hype about prediction markets was based on the fact that InTrade has been claiming that their exchange did forecast “all 50 states” right in the 2004 US presidential elections.
-
While at the same time, Nate Silver (aggregator of polls, polling model) has become a modern hero.
http://www.midasoracle.org/2009/02/12/nate-silver-is-the-15-web-celebrity/
-
Which is why I think it was a strategic error to put the emphasis on accuracy. The added accuracy is minimal (and even this is being questioned by some researchers, see the columbia link above). Who really cares about +2.7%?
-
On the other hand, an argument based on a certain aspect of efficiency (how fast facts and expertise are incorporated) is always a winner in the complicated situations. Now, the hard part is to spot those “complicated situations”, and to document the comparison between the prediction markets and the mass media. It will be costly —$$$.
-
“The key value in any type of prediction is accuracy – how can an inaccurate prediction, even delivered early be of value?”
-
Matt,
The issue is not about inaccuracy, but about the very minimal level of added accuracy brought by the prediction markets. Why would people care about political prediction markets if Nate Silver (and all the other poll aggregators) nail the elections?
-
“That our simple prediction market tools can come anywhere close on accuracy is a huge credit.”
Nothing new for Hayek:
http://www.econlib.org/library/Essays/hykKnw1.html
-
“In myriad examples – Intel, HP, EA etc, a well designed prediction market has delivered significantly more accurate forecasts.”
-
Matt,
I don’t think that, in all honesty, you can talk about “myriad” examples. There is one or two couple(s) of cases, and most of them are not well documented. And the “flow of information” case was not about PMs but about a peripheral issue.
-
If I were involved with a corporate PM consultancy, I’m not sure how much these short-term “marketing”-type issues that Chris brings up would bother me, though now I would certainly steer towards framing things in terms of competitive forecasting/averaging as opposed to the stock market metaphor.
In terms of demonstrating the value of these mechanisms, I am more bullish on the corporate context in the near future than the real-money exchanges. At least corporate PMs have relatively high turnover and tend to have fixed trading budgets. There needs to be a good way of evaluating their relative contribution of course.
Incidentally, I notice that Nate Silver has made some Oscar predictions:
http://nymag.com/movies/features/54335/
http://www.ritholtz.com/blog/2009/02/the-velocity-of-prediction-markets/
I cannot put it better than Panos Ipeirotis on accuracy vs efficiency, and I do agree with many of the “radical” comments above.
Here’s what I consider the main benefits of running a marketplace:
- bringing people together
- fostering conversations
- summing up information
- transmitting it through prices
Flubber is partially right.
http://www.midasoracle.org/2009/02/18/flubber-velocity-inaccuracy-prediction-markets/
Whatever the expectations (whether they will end up being wrong or right), in complicated situations, the prediction markets aggregate them faster than the mass media do.
http://predicts.betfair.com/2009/02/prediction-market-theory-is-it-accuracy-or-efficiency-that-is-most-important-when-it-comes-to-prediction-markets/
http://betting.betfair.com/specials/politics-betting/prediction-markets/prediction-markets-is-it-accuracy-or-efficiency-that-is-most-180209.html
Velocity is such a potent argument. Why don’t we use it more, for Christ’s sake?
http://www.midasoracle.org/2009/02/18/velocity-prediction-markets/
The very last line of the Erikson–Wlezien paper
http://www.midasoracle.org/2009/02/18/erikson-wlezien-prediction-markets/
The HHS–Sebelius prediction market might be (yet) another case-in-point for documenting velocity.
http://www.midasoracle.org/2009/02/19/kathleen-sebelius-health-and-human-services/
Prediction markets didn’t revolutionize decision making —and will never do. However, they are a nice complement to the classic forecasting toolkit.
http://www.midasoracle.org/2009/02/19/prediction-markets-didnt-revolutionize-decision-making/
JFK & The passing of time & The prediction markets
http://www.midasoracle.org/2009/02/20/the-passing-of-time/
Velocity + Inaccuracy
http://www.midasoracle.org/2009/02/22/velocity-inaccuracy/
I created a tag, “The Truth About Prediction Markets”, so that people reading the post above can follow up with the latest on this same topic:
http://www.midasoracle.org/tag/the-truth-about-prediction-markets/
Accuracy and Efficiency of Prediction Markets
http://www.midasoracle.org/2009/02/23/accuracy-and-efficiency-of-prediction-markets/
HubDub takes home the Gold at the Oscars
http://www.midasoracle.org/2009/02/23/hubdub-takes-home-the-gold-at-the-oscars/
Criticism, here:
http://www.midasoracle.org/2009/02/23/hubdub-takes-home-the-gold-at-the-oscars/#comment-23480
“It is their velocity that we should put to work.”
http://www.midasoracle.org/2009/02/25/it-is-their-velocity-that-we-should-put-to-work/
The Economist: The enterprise prediction markets are flopping, big time.
http://www.midasoracle.org/2009/02/26/the-economist-enterprise-prediction-markets-novel-way-generating-forecasts-yet-to-take-off/
An uncertain future – A novel way of generating forecasts has yet to take off. – by The Economist – 2009-02-26
http://www.economist.com/business/displaystory.cfm?story_id=13184829
How vendors are scuttling the field of enterprise prediction markets —and the prediction market industry, as a whole
http://www.midasoracle.org/2009/02/27/vendor-conferences-without-any-editorial-line/
6 years is long enough for an assessment of the prediction markets.
http://www.midasoracle.org/2009/02/28/6-years-is-long-enough-for-an-assessment-of-prediction-markets/
Do economics professors who have business relationships with prediction market companies indoctrinate their students about the supposed benefits of prediction markets?
http://www.midasoracle.org/2009/03/03/economics-professors-indoctrinate-their-students-about-prediction-markets/
Opacity versus Openness
http://www.midasoracle.org/2009/03/05/opacity-versus-openness/
Enterprise prediction markets… the next big thing —not.
http://www.midasoracle.org/2009/03/06/the-economist-prediction-markets-niall-o-conno/
Dumped polls outperform the prediction markets.
http://www.midasoracle.org/2009/03/07/dumped-polls-outperform-the-prediction-markets/
How that prediction market consultant in Holland attracts economic advisers on the cheap
http://www.midasoracle.org/2009/03/07/how-that-prediction-market-software-vendor-in-holland-attracts-economic-advisers-on-the-cheap/
http://marketdesigner.blogspot.com/2009/03/prediction-markets-why-arent-they-used.html
http://blog.mercury-rac.com/2009/03/08/the-value-of-prediction-markets/
Will prediction markets ever really take off?
http://www.midasoracle.org/2009/03/15/will-prediction-markets-ever-really-take-off/
Will enterprise prediction markets ever really take off? — Now, *my* answer.
http://www.midasoracle.org/2009/03/25/the-economist-enterprise-prediction-markets/
http://torontopm.wordpress.com/2009/04/05/practical-enterprise-prediction-markets/
http://blog.mercury-rac.com/2009/04/04/follow-up-to-approaching-business-problems-differently/
an Analysis of HP’s Real Prediction Markets
http://torontopm.wordpress.com/2009/04/12/an-analysis-of-hps-real-prediction-markets/
The truth about (enterprise) prediction markets
http://www.midasoracle.org/2009/04/13/the-truth-about-enterprise-prediction-markets/
Gartner: The “benefit” of enterprise prediction markets is “moderate” and “early users, who have begun to overestimate their accuracy and overall usefulness, are now somewhat disillusioned with the technology.”
http://www.midasoracle.org/2009/04/15/gartner-enterprise-prediction-markets-2008/
Are prediction markets useful to our global civilization?
http://www.midasoracle.org/2009/05/13/prediction-markets-5/
What Robin Hanson didn’t tell you about collective forecasting
http://www.midasoracle.org/2009/05/26/what-robin-hanson-didnt-tell-you-about-collective-forecasting/
“Tap The Collective†failed to convince me about the usefulness of enterprise prediction markets
http://www.midasoracle.org/2009/09/05/tap-the-collective-fail/
Do businesses need enterprise prediction markets?
http://www.midasoracle.org/2009/09/08/do-businesses-need-enterprise-prediction-markets/
Midas Oracle debunks the EPM software vendors’ shallow marketing discourse. But there is a price to pay, of course.
http://www.midasoracle.org/2009/09/16/debunking-shallow-marketing-discourse-retaliation/
The truth about CrowdClarity’s extraordinary predictive power (which impresses Jed Christiansen so much)
http://www.midasoracle.org/2009/09/29/crowdclarity-general-motors/
The Chicago candidacy to host the Olympics, which was favored by the prediction markets (and gullible bettors like Ben Shannon), is the one that fared the worst.
http://www.midasoracle.org/2009/10/03/assessing-prediction-markets/