I raved the other day about the coming of search engines for (public) data, and I dreamed up about search engines for prediction market data. What I completely forgot to mention is that most of prediction exchanges don’t publish their archives on the Web. Most prediction exchanges (but HubDub) make it impossible for any search engine to spider and index their archives. We are still living in the pre-historic age, folks —for the good reason that most exchange executives don’t understand the Internet and don’t get how to market on the Web.

-

-

My review of the literature and case studies (that have been published) indicates that prediction markets have improved the accuracy of forecasts, but the improvements have not been great enough to encourage widespread (or even minimal) acceptance. Furthermore, these studies like to average their results over a number of markets, disguising the fact that some markets improve forecasts, while others fail to do so. Some studies look at average absolute errors, covering up the fact that some predictions were underestimating the true outcome and others overestimating it. This means the real errors are as much as twice as large as those reported. Few, if any, explanations for the failures are ever presented. This raises the issue of consistency. In case studies such as these, where there is no clear under- or over-estimation tendency, for which a correction may be made, the prediction errors are just too great.

Clearly, if similar prediction markets do not provide consistently accurate forecasts, they will not be relied upon for any important business decisions.

[...] When it comes to the first point, forecasting something that the company already forecasts, prediction markets may or may not be an excellent solution. I’ve seen one set of markets that absolutely blew away the accuracy of current forecasts, and I’ve seen other markets that were consistent with current forecasts with little or no accuracy edge. [...]

Care to say more about what is the determinant of an EPM success?

Strangely, Chris’ writings on prediction markets have become quite negative in recent months.

I am actually trying to be more realistic than the crowd.

Jed, if you read closely the intro on top of the blog frontpage, you will see that I am actually bullish on some applications of prediction markets.

My other response to Chris is that I feel he doesn’t distinguish between public and internal prediction markets when he discusses marketing and usage of these markets.

I am well aware of the 2 completely different landscapes, Jed, but I wanted a statement that could fit both sub-fields. Thanks for your comments. I will think about all this…

I warned my readers many times against the vendor conferences organized by the San Francisco man. He is so desperate that he invites anybody who will pronounce the word “prediction” and “markets” in the same paragraph. Many of the invited speakers haven’t the slightest knowledge of the field of prediction markets. As for the vendors, they are incapable of producing one single case study featuring a success in the use of enterprise prediction markets. Not a single one. (And I won’t mention the “flow of information” —the worst research ever published on prediction markets.) Their vendor websites publish lists of clients, which, at first glance, look impressive, but many of those so-called customers are in fact ancient clients who have ended pilot programs years ago. To add insult to injury, this fake conference is sold $400 to gullible attendees. It is not even worth 4 cents.

The Economist reporter who attended the San Francisco conference realized what I [*] realized long ago: The field of enterprise prediction markets is all smokes and mirrors. The more the prediction market vendors will participate in such crappy conferences, the more the media will realize that the prediction market vendors are all hat and no cattle, and the more they will publish news stories bursting the prediction market bubble. And in the end of 2009, we will end up with 10 news articles in major media telling the world that prediction markets were a fad. Live by the hype; die by the hype.

The only way to get out of this debacle is to come back to basics: Do the research right, do discover the real value of enterprise prediction markets (velocity), and, then, only when you have something to show for it, go out in postings and conferences.

-

[*] I follow the field of prediction markets since 2003. I saw it in all shapes and stripes. You can fool your mother, but you can’t fool me.

-

APPENDIX:

An uncertain future – A novel way of generating forecasts has yet to take off. – by The Economist – 2009-02-26

- But although they have spread beyond early-adopting companies in the technology industry, they have still not become mainstream management tools. Even fervent advocates admit much remains to be done to convince sceptical managers of their value.

- Koch says the results so far have been pretty accurate compared to actual outcomes, but stresses that markets are complementary to other forecasting techniques, not a substitute for them.

- A big hurdle facing managers using prediction markets is getting enough people to keep trading after the novelty has worn off.

- Another reason prediction markets flop is that employees cannot see how the results are used, so they lose interest.

- Bosses may also be wary of relying on the judgments of non-experts.

-

- But although they have spread beyond early-adopting companies in the technology industry, they have still not become mainstream management tools. Even fervent advocates admit much remains to be done to convince sceptical managers of their value.

- Koch says the results so far have been pretty accurate compared to actual outcomes, but stresses that markets are complementary to other forecasting techniques, not a substitute for them.

- A big hurdle facing managers using prediction markets is getting enough people to keep trading after the novelty has worn off.

- Another reason prediction markets flop is that employees cannot see how the results are used, so they lose interest.

- Bosses may also be wary of relying on the judgments of non-experts.

My remarks:

1. The recent San Francisco conference (where the reporter from The Economist got his/her tidbits) has been counter-productive. No wonder, since that conference has no editorial line (it is a vendor conference). I told you so.
2. The hype about (enterprise and public) prediction markets is not substantiated. I told you so.

-

Following the arrest (and assassination) of Lee Harvey Oswald, a number of people advocated his innocence —and, instead, claimed that the death of JFK was the result of a CIA–FBI–Military conspiracy. A short list of conspiracy theorists goes like this:

- Lee Harvey Oswald himself (“I am just a patsy!“);

- His mother (who claimed that her son actually worked for the FBI);

- Mark Lane (Oswald’s post-mortem attorney);

- Colonel L. Fletcher Prouty (who fed Jim Garrison with conspiracy theories);

- Jim Garrison (who unsealed the Zapruder film);

- Many commentators who interpreted the Zapruder film as showing that one bullet came from the “grassy knoll” (situated in front of Kennedy, on the right side of the road), since JFK’s head went violently backward;

- and, of course, Oliver Stone.

-

At least 3 facts infirm the hypothesis that Lee Harvey Oswald was either a member of a conspiracy or a decoy aimed at distracting attention from the real conspirators (and, hence, also a victim of that conspiracy):

1. We know now that Oswald tried to kill the General Walker (of the John Birch Society), some months before the JFK assassination. He also had views on Nixon. He failed with Walker and Nixon, but succeeded with Kennedy.
2. The fact that JFK’s head went violently backward is a normal nervous reaction from someone shot in the head from behind.
3. 45 years after the JFK assassination, supposedly plotted by the top brass at the FBI, the CIA and the Military, nobody has ever come forward with a tell-all story that could make him/her filthy rich thru books, movies, and merchandising.

-

The key point is the passing of time.

The conspiracy theories didn’t pan out.

-

The (contemporary) prediction markets appeared on the scene in 1988. Since 2003 (with PAM) and 2004 (with “The Wisdom Of Crowds”), the mass media and the vertical media have published many stories about the predictive power of the prediction markets (quoting Robin Hanson and Justin Wolfers), how “impressive” and fascinating all this is, and how numerous could be the applications of this new forecasting tool.

In 2009, the reality check is that it didn’t pan out. The public prediction markets are far from being “impressive”, and no success story has ever been published about the enterprise prediction markets.

We need to reset and reboot the field of the prediction markets.

-

Previously: The truth about prediction markets

In the conclusion of their introduction, Robert Hahn and Paul Tetlock wrote that they want their readers to contemplate the idea that prediction markets could make a “big” difference and “revolutionize public- and private-sector decision-making”. Well, 4 years later, it is clear that those big dreams didn’t pan out. Not a single mass media outlet has praised the public prediction markets for their work on the 2008 US presidential election (I am taking about a post-mortem analysis about Election Day, not the primaries). Not a single one. (Not even Justin Wolfers.) And the number of corporations using enterprise prediction markets is still minute. The thinkers who wrote this book (“Information Markets”) all made the mistake to put the emphasis on accuracy instead of efficiency. That was the foundation flaw. We should reset and reboot the field of prediction markets.

Previously: The truth about prediction markets