[...] In virtually every case, the prediction market forecast is closer to the official HP forecast than it is to the actual outcome. Perhaps these markets are better at forecasting the forecast than they are at forecasting the outcome! Looking further into the results, while most of the predictions have a smaller error than the HP official forecasts, the differences are, in most cases, quite small. For example, in Event 3, the HP forecast error was 59.549% vs. 53.333% for the prediction market. They’re both really poor forecasts. To the decision-maker, the difference between these forecasts is not material.
There were eight markets that had HP official forecasts. In four of these (50%), the forecast error was greater than 25%. Even though, only three of the prediction market forecast errors were greater than 25%, this can hardly be a ringing endorsement for the accuracy of prediction markets (at least in this study). [...]
To the despair of this man, Paul’s analysis is quite similar to mine (circa February 14, 2009):
The prediction market technology is not a disruptive technology, and the social utility of the prediction markets is marginal. Number one, the aggregated information has value only for the totally uninformed people (a group that comprises those who overly obsess with prediction markets and have a narrow cultural universe). Number two, the added accuracy (if any) is minute, and, anyway, doesn’t fill up the gap between expectations and omniscience (which is how people judge forecasters). In our view, the social utility of the prediction markets lays in efficiency, not in accuracy. In complicated situations, the prediction markets integrate expectations (informed by facts and expertise) much faster than the mass media do. Their accuracy/efficiency is their uniqueness. It is their velocity that we should put to work.
Prediction markets are not a disruptive technology, but merely another means of forecasting.
Go reading Paul’s analysis in full.
I would like to add 2 things to Paul’s conclusion:
- We have been lied to about the real value of the prediction markets. Part of the “field of prediction markets” (which is a terminology that encompasses more people and organizations than just the prediction market industry) is made up of liars who live by the hype and will die by the hype.
- Prediction markets have value in specific cases where it could be demonstrated that an information aggregation mechanism is the appropriate method that should be put at work in those cases (and not in others). Neither the Ivory Tower economic canaries nor the self-described prediction market “practitioners” have done this job.
Hi Chris…
We should keep in mind that the HP test prediction markets were run over a decade ago. I would think that we have a better understanding of prediction markets, today. Their constraints may have been more onerous than they disclosed in their paper. They confined the trials to a small number of participants, and it appears they were selected mainly from the marketing functional area. Maybe their definition of diversity was much too narrow.
The authors don’t tell us which products’ sales were being forecast. It could be that the markets with the largest errors were those for relatively new products (much more difficult to forecast market acceptance). It would have been nice to know this, but they were probably restricted in the information that could be disclosed in the paper.
Still, the paper was a good start. We did learn a great deal from their trials. It is a shame that there are so few published examples. There is a great opportunity to learn from each other and improve the forecasts for everyone.
I happened to be on Inkling’s web site to see what information they provide about their corporate case studies (http://inklingmarkets.com/homes/howtouse). Even though there aren’t many (9), most of them are still running up to two years later. Not much information is provided as to the usefulness or accuracy of the predictions, but the fact they’re still running indicates that they must be useful, given the cost of running the markets.
What is really interesting is the number of participants in the markets – they all had substantially more participants than any published prediction market. One had 5,000 and two others were over 1,000 participants. Even the market with the fewest traders still had 220 – far more than the small numbers used in other published markets. And, as far as we know, these markets would have utilized an automated market maker.
The conclusion is that, even when liquidity is not expected to be an issue, these corporate decision-makers decided (correctly) to utilize a substantial number of traders to ensure that they would get the best possible results. It would be very interesting to see how accurate their predictions were.
The HP paper took place a decade ago with a handful of people and a handful of markets. I would hardly extrapolate any major conclusions from that across the last ten years of PM development!
Regarding # of participants in a market, that can be interpreted in many different ways. Was it the number of people that had an account in the market, or the number of people active? Of the people with an account, was it registered for them (aka from a corporate directory automatically) or did they have to take action? And then, how many people were trading in each market?
And Inkling does use an automated market maker; virtually any corporate PM has to for UI reasons.
Again, I found in my research that as few as 16 people actively trading in a market can produce calibrated results. (Where a 20% estimate occurred 20% of the time, etc.) Sure, more people is always better but you can still get good results from a small active trader base.
The bulk of Paul’s HP analysis is timeless. You would analyze today’s EPMs and would end up with the same conclusions. Do transfer some modern-day data to Paul, and do ask him to look into them, and let’s see.
Jed, I was looking into the, admittedly old, study on HP’s markets, because they continue to be referenced (positively) by just about every consultant, academic, presenter and journalist, when talking about prediction markets. Once it gets repeated enough, it becomes “true”. I just decided to go back to the paper and look at the numbers myself.
The authors of the paper did come to conclusions regarding those prediction markets. I’m not faulting them for having done so, maybe I’m just saying they should have reported them as “observations” rather than conclusions.
As for the Inkling corporate case studies, I’m sure that not all of the participants really participated, but it does indicate that they believed they would need a large number of people to arrive at useful results (or at least a reasonable level of trading). I have followed your research (at least in basic terms), but I would be concerned with the consistency of the predictive accuracy of small groups. Given the relatively low cost of using additional participants in these markets, why focus on using fewer traders (and run the risk of losing “diversity” in the crowd)?
Believe it or not, I do think that prediction markets can be accurate and useful in corporate decision-making. It would be nice to have more data on actual applications, so that everyone could learn more about what works and what does not.
Regarding low numbers in prediction markets, I agree that it’s always better to have more people trading than less. The point of my studying was to try and put a lower bound on the number of people needed for a calibrated forecast. In some situations, getting additional participants in markets is actually *very* costly.
I may sound like a broken record, but accuracy is only one of many potential benefits of prediction markets. Sometimes it’s a LOT better than current forecasts (I’ve seen a case of error rates getting cut in half), and other times it is only a marginal increase. That has to be balanced with the costs of forecasting errors, which may mean that even a 1% increase is worth the costs.
I do hope that more papers with real-world case studies are written; unfortunately many companies still don’t want to release any of that information.
“unfortunately many companies still don’t want to release any of that information.”
The PM industry could set up private prediction markets pertaining to the industry, and each PM operator, and then release the data for everyone to see.
I know that it can be very costly to add large numbers of participants, but unless there is a sufficient number, the mechanics of the aggregation method break down. Then, you are more likely to have inconsistent results. As you noted, Jed, sometimes the predictions are a lot better and sometimes they’re just marginally better. Sometimes they’re *worse* than the current forecast. The problem is we don’t know which is going to be true, and that is one of the things we’re trying to determine with prediction markets.
Sometimes a 1% improvement in a forecast is valuable, other times it’s not. Sometimes a 5-10% improvement will be irrelevant (i.e. if you’re still out by 40-50%). It depends on the issue being predicted.
Maybe we should be trying to get these prediction markets working as well as possible before we try to make them more cost effective (with fewer participants). As it stands, apart from generalities, we really don’t know what it takes to make prediction markets *consistently* useful predictors of the future.
I think one of the hangups we get when talking about prediction markets is comparing them to other forecasting methods. Yes, that’s one measurement of success, but it’s rather limiting.
Some of the more interesting markets are on topics that were previously thought to be “un-forecastable.” (I often ask people to think about PM’s on project management RAG status.) There’s nothing to compare a PM forecast to (other than reality), so any decent result is a 100%+ improvement! Todd Proebsting’s first PM at Microsoft is a classic example of this exact idea.
In my experience, there are a LOT of factors that go into making a successful PM. I’ve run PM’s where I got hundreds of people to register without any promise of even a cash or gift reward and got great results. I’ve run PM’s where it was pulling teeth to get 20+ people involved (from a list of several hundred) when they were getting paid to participate. As always, I like to refer to what Emile of NewsFutures talks about: Rewards, Recognition, and Relevance. There’s no “magic mix” of these, but fundamentally that’s what the success of each market comes down to. (NOT the number of traders…)
Getting a good answer only works if you’re asking a sensible question to a sensible group of people. To me, PM’s have the most difficultly with the sensible question aspect of the equation.
I couldn’t agree more on the need for sensible questions, whose answers are in need of prediction. Several conference presenters and papers have stressed this, but no one goes into the details as to how to select appropriate questions. Maybe we should focus on that.
I think we would all be interested in your comments on what did and did not work in motivating trading behaviour in the actual PMs that you operated. Let’s look at the data and get as many answers as we can. There may not be hard and fast rules, but there may be approaches that are more likely than others to succeed.
One thing is for sure. There is a great hunger, here, for practical advice on how to operate effective prediction markets. I mean details, not generalities. There is, however, a severe lack of such advice.
I applaud your efforts to operate markets with the fewest number of active participants, but, I can’t resist, I still think you need larger groups to ensure adequate diversity within the crowd. I’ll keep an open mind, however.
Gartner: The “benefit” of enterprise prediction markets is “moderate” and “early users, who have begun to overestimate their accuracy and overall usefulness, are now somewhat disillusioned with the technology.”
http://www.midasoracle.org/2009/04/15/gartner-enterprise-prediction-markets-2008/
In response to Paul, I think that it’s very difficult to discuss these factors in a meaningful way. It’s almost like trying to answer the question “What makes a good piece of art?” Each prediction market is unique based on what it’s trying to accomplish, with whom its trying to accomplish, and the tools used to do that.
Unfortunately most of the projects that I’ve worked on are covered under NDA’s, so I can’t release data or findings from them. I would assume this is likely the case for most of the PM software vendors.
I think more and more of the PM software companies are going to be providing deeper consulting services specifically because of this; they’re the central node of knowledge and experience across many prediction markets. Not only do their clients purchase their software, but they often need to use their consulting services to help make sure the markets run as best as they can.
Hi Jed…
I am well aware of the NDAs and other confidentiality issues surrounding corporate forecasts, but surely the data can be disclosed in a way that protects corporate interests, yet helps in our analyses of prediction markets. There needs to be some openness about prediction markets, otherwise we’re limited to a few published results with claims that cannot be verified. I guess it’s the accountant in me talking.
As for discussing the factors that make a prediction market work effectively, If we aren’t able to do so, and prediction markets become too subjective, the corporate world will never take to them. No one questions that forecasting is part “science” and part “art”, but when it gets too arty, it will scare the corporate world. We need more “science”, more data (even anonymous data).
- Data could be anonymized.
- The PM consultants could set up and run open cases (pertaining to themselves, the prediction markets industry, or any other public topic) just for the purpose of reporting the results to the public.
There are ways to lift the secrecy veil. If the consultants were truthful and if EPMs really provided positive benefits, their advocates would easily find ways to put the good results under people’s nose.
The reason why there is not “openness” is that EPMs do not work in the way they are hyped.
In our quest how to make PM’s more accurate, I would like to focus on two forces, briefly.
-
1. A market gets (a little) closer to “true” value when a participant comes along and adds something of value, informational value, that hasn’t already been digested by the market to the extent it should have.
Such a person is likely to have done some research or at least collected specific pieces of information somewhere. For this person to spend energy on doing research there has to be an incentive, a big one. If that person is offered the opportunity to make $10 or even $50 it is very unlikely that he will waste much time on doing research. He will probably bet whatever the stake is and guess… Well, PM’s don’t outperform other methods by collecting hunches or votes.
When people are offered the opportunity to make $1000 or more, believe me, they will do their homework. So, basically we don’t want to invite all the “experts”, we need to create them, and as such, because people are so different, every one of them will have their own approach and strategy. It is this diversity that is very important, IMO.
2. Big money more often than not has superior information behind it. This is very important when we want market prices to approach “true” probability. When every participant is given an equal amount to play with it seems to me we are missing a crucial aspect.
So, ik keep going back to an open real money exchange as a way to improve the the value of PM’s.
Medemi, I like the way you think. Yes, most people have some piece of information that needs to be aggregated. Sometimes their “research” is simply their unique perspective or observation. When the incentives (and penalties for wrong guesses) are too low, there will be too much guessing. Diversity is the key.
“Big money” comes from better “investing”. Even when the wealth is distributed equally at the beginning, it won’t take long for the better investors to have more clout than the “ignorant” guessers (who simply provide tiny bits of information and lots of market liquidity). There was a paper written that touches on this: Suckers Are Born but Markets Are Made: Individuals, Rationality, Arbitrage, and Market Efficiency on an Electronic Futures Market, by Kenneth Oliven & Thomas A. Rietz. I think there is a link to it on this Midas Oracle.
I do believe that real money will improve traders’ investment decisions, but I don’t think it needs to be large amounts of money. When I play golf matches, it isn’t the amount of money won, it is the fact that I won that counts. Even if it’s only a $5 Nassau. It’s the bragging rights. In the corporate setting, eventually superior predictive abilities can be rewarded via compensation plans (which could amount to thousands of dollars, annually). Another option might be to use “points” that could be redeemed for merchandise, flights, etc… This *may* get around the anti-gambling laws that restrict US prediction markets (and others).
16 comments already… Wow… Paul, you have a knack at suscitating the debate…
“What makes a good piece of art?â€
If it is that subjective, then why bother?
>> “16 comments already… Wow… Paul, you have a knack at suscitating the debate…”
Every experienced player would instantly recognize Paul’s quality insights.
Paul, yes, you filled the gap I left behind and made it more practical.
These companies shouldn’t go cheap on us now… A $5,000 first prize would be a wise investment for any company that cares about accurate predictions. With that kind of prize money 30 or 40 traders would probably do the trick. Get them trading simultaneously in a narrow window on a daily basis and you won’t even have a liquidity problem.
The more I think about it, the less impact I feel guessers have on accurate markets. Every full-timer I’ve spoken to (most of them traders) has said that this lifestyle requires a lot of hard work. These are usually the kind of people who don’t want to be treated as “slaves” in the corporate setting, or who felt they didn’t get the respect they deserved.