To recap, the prediction market beat the official GM forecast (made at the beginning of the month) easily, which isn’t hugely surprising considering the myopic nature of internal forecasting. But the prediction market also beat the Edmunds.com forecast. This is particularly interesting, as Edmunds would have had the opportunity to review almost the entire month’s news and data before making their forecast at the end of the month. [...]
Assume that even with three weeks’ early warning Chevrolet was only able to save 10% of that gap, it’s still $80million in savings. Even if a corporate prediction market for a giant company like GM cost $200,000 a year, that would still be a return on investment of 40,000 %. And again, that’s in the Chevrolet division alone. [...]
Make up your own mind by reading the whole piece.
Next: Assessing the usefulness of enterprise prediction markets
At first blush, it appears that we finally have a bona fide prediction market success! If we’re going to celebrate, I’d suggest Prosecco, not Champagne, however.
There are a number of reasons to be cautious. These represent only a couple of markets. We don’t know why Urban Science people appear to be so adept at forecasting GM sales in turbulent times. There is no information on the CrowdClarity web site to indicate why the markets were successful nor how their mechanism might have played a role in the PM accuracy. I’m guessing that it would have been really easy to beat GM’s forecasts in November, as they would likely have been even more biased than usual, mainly for political reasons. I’m not sure how Edmunds.com’s may have been biased or why their predictions were not accurate. Maybe they are not so good at predicting unless the market is fairly stable.
The CrowdClarity web site boasts that a few days after the markets were opened, the predictions were fairly close to the eventual outcome. This is a good thing, but, at this point it is not useful. No one knew, at that time, that those early predictions would turn out to be reasonably accurate. As a result, no one would have relied upon these early predictions to make decisions.
I’m even more skeptical of the company’s contention that markets can be operated with as few as 13 participants. Here we go again, trying to fake diversity.
It is interesting that a prediction market comprised of participants outside of the subject company did generate more accurate predictions than GM insiders (biased) and Edmunds.com (experts). The question that needs to be answered is why. Clearly, Urban Science people did have access to better information, but why?
Unless we know why the prediction markets were successful at CrowdClarity, it is hard to get excited. There are too many examples of prediction markets that are not significantly better than traditional forecasting methods. This one could be a fluke.
I’ll have more to say, soon, when I write about the prediction markets that were run at General Mills. There the authors of the study found that prediction markets were no better than the company internal forecasting process.
Paul, quick question. Have you ever administered/run a prediction market?
Jed, a quick question. How is your question relevant to the issue at hand? The issue is to determine whether the enterprise prediction markets are actually useful to businesses. Paul Hewitt is perfectly qualified to assess that, independently from the lobbying from the EPM software vendors.
Chris, it was a question for Paul. I’m just curious if he’s administered/run a prediction market.
Regarding your comment, what is your standard for being “perfectly qualified to assess” prediction markets? I just want to understand what you consider important.
Jed, Paul is into information economics. Read the archives of Midas Oracle.
Chris, that wasn’t my question. What is *your* standard for someone to be “perfectly qualified to assess” prediction markets?
Like I said, I just want to understand what you consider important in order to do that.
Hi Jed…
As for qualifications, I have been making business decisions for almost 30 years. I am a chartered accountant and a business owner. Starting in university and continuing to this day, I have been researching information needs for corporate decision making. As Chris points out, I’m not a salesperson for any of the software developers. In fact, if I have a bias, it is to be slightly in favour of prediction markets. That said, I still haven’t seen any convincing evidence that they work as promised by ANY of the vendors.
As for whether I have ever run or administered a prediction market, the answer is no. Does that mean I am not qualified to critique the cases that have been published? Hardly. You don’t have to run a PM to know that it is flawed. Those that do, end up trying to justify minuscule “improvements” in the accuracy of predictions. They also fail to consider the consistency of the predictions. Without this, EPMs will never catch on. Sorry, but that is just plain common sense.
The pilot cases that have been reported are pretty poor examples of prediction market successes. In almost every case, the participants were (at least mostly) the same ones that were involved with internal forecasting. The HP markets, yes, the Holy Grail of all prediction markets, merely showed that prediction markets are good at aggregating the information already aggregated by the company forecasters! They showed that PMs are only slightly better than other traditional methods – and mainly because of the bias reduction. Being slightly better is not good enough in the corporate world.
I think I bring a healthy skepticism to the assessment of prediction markets. I truly want to believe, but I need to be convinced. I am no evangelist, and there is no place for that in scientific research. Rather than condemn me for not administering a PM, why not address the real issues that arise from my analyses?
Paul, I was just asking a simple question… not attacking or “condemn”ing you!
I simply didn’t know if you were taking your analysis from the papers that have been published or also your own experiences.
My question for Chris is to understand what standard he considers an “expert”. (ie, academic credentials, blog Google PageRank, industry experience, etc.) That’s mainly because I think his standards vary depending on who he wants to criticize on any given day.
Jed, the fact that I am unpredictable from your perspective does not mean that I am erratically irrational.
Hi Jed…
Clearly my wording was a bit harsh. Sorry. I understand that operating a prediction market will give you some practical experience that will improve their functionality in the future. I suppose my point is that I don’t need this experience, right now, to critique the existing pilot projects, because they are failing at the theoretical level (at least they aren’t “succeeding” by any significant measure).
Given the relative silence from the academic community (lately), I think that someone with my background can contribute to the discussion, especially with respect to decision making and the role of information in the process. In order for this mechanism to take hold, it has to be proven that it works (and under which conditions).
Hi, Paul.
I agree that CrowdClarity’s slides don’t get into the detail necessary to understand why they were successful. But that’s largely because those slides were published as a sales tool… a top-line attention grabber for potential sales leads.
The things I would want to understand to truly evaluate success would include: demographics of traders, the types of forecasts that already exist, what error rates are when sales volatility is more “normal”, what trader incentives were used, etc.
Regarding your point on number of traders, my own research showed that as long as you have more than 15 traders, the market generates a calibrated result. (The markets I ran were probabilistic.) So that’s actually quite a realistic number, even though it might intuitively seem low. But I would point out that because of the nature of the markets I ran, the likelihood that traders knew each other was low, so there was a natural diversity in the 15+ that I studied.
I would certainly consider the prediction market / case study CrowdClarity published a “success.” The key for me is turning that into a *business* success. You’re very right that if this was one of the first months that a PM was run, no one would have likely believed the results, even if they were the closest to the eventual truth.
But let’s say that a PM has been showing reasonable accuracy for several months, which I would define as similar or better accuracy than other forecasting methods. Then, like the case study, the prediction market shows a drastically different result than any other forecast. While a manager probably won’t “bet the farm” on a prediction market alone, that certainly would warrant re-thinking the forecast.
The reason is simply that the market aggregates the information more quickly than other methods. I would argue that GM and Edmunds.com used forecasting models that are quite good most of the time, but completely wrong when some of the basic assumptions collapse. Since the PM doesn’t rely on algorithms, collapsing assumptions won’t affect accuracy.
In many ways, I think this last point is the most important with prediction markets. 80-90% of the time a prediction market might generate a forecast with accuracy that’s on par with other methods… maybe a little better, maybe a little worse. But the other 10-20% of the time, when the forecasts diverge significantly, is where prediction markets can be *very* useful. When assumptions behind traditional models weaken or collapse, a prediction market can be the early warning signal, since it uses a different methodology to generate a forecast.
Maybe a company spends $3-4k a year on a prediction market that generally confirms or better brackets existing forecasts 11 months out of the year. But if the intelligence it generates that 12th month of the year helps the company save $20k, it strikes me as a wise investment.
Again, that’s not to say that a prediction market is going to solve a company’s problems. It needs to address problems where the cost of the error is worth the investment and where a prediction market can effectively address it. And that’s a sensitive balance.
Hi Jed…
I agree with you about some of the possible reasons that the prediction market was better than traditional forecasting methods. Edmunds.com probably does use an algorithm to come up with their guess, and when the market undergoes a fundamental change (like it did), the algorithm will not be able to generate an accurate prediction. Consequently, the prediction market becomes a very good alternative. The problem is that you need to be able to trust the prediction market will be more accurate than the algorithmic method. This means we need more and better research about prediction markets. In particular, we need to know which types of things can be consistently predicted accurately and why they often fail.
Prediction markets (assuming they are consistently accurate) can complement traditional forecasting by providing the distribution of guesses around the mean. Very useful for risk analysis and contingency planning. Wouldn’t you agree, however, that the academics that were developing the concept have been pretty silent of late?