Another tried-and-true method is to repeat a statement over-and-over until it becomes a “fact”. The best example of this, regarding prediction markets, is the claim that they are “proven” by virtue of HP’s remarkably lop-sided, unmistakable, trouncing of internal forecasts by a score of 6 – 2. It is usually cited as “6 out of 8″ or 75% better than the official forecasts. The “truth” is that the “wins” were by minuscule margins and many of the forecasts were equally bad. [...]

Read the rest, here.

External Link: Paul Hewitt’s blog on prediction markets

IIF’s SIG on Prediction Markets

Research Papers

Basics

Several studies explain the concept of prediction markets and provide useful summaries of the method, e.g.

- Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]
- Wolfers, J. & Zitzewitz, E. (2006). Prediction Markets in Theory and Practice, New Palgrave Dictionary of Economics and the Law (in press). [Full text]
- Wolfers, J. & Zitzewitz, E. (2004). Prediction Markets, Journal of Economic Perspectives, 18, 107-126. [Full text]
- An overview and classification of 152 studies on prediction markets, published between 1991 and 2006, is provided by
Tziralis, G. & Tatsiopoulos (2007). Prediction Markets: An Extended Literature Review, Journal of Prediction Markets, 1, 75-91. [Full text]

Evidence on the accuracy of prediction markets

This section summarizes research that analyzes the relative performance of prediction markets and other forecasting methods.

Markets vs. polls (election forecasting)

- Berg, J., Nelson, F. & Rietz, T. (2008). Prediction Market Accuracy in the Long Run, International Journal of Forecasting, 24, 283-298. [full text]
- Erikson R. S. & Wlezien C. (2007). Are Political Markets Really Superior to Polls as Election Predictors? Public Opinion Quarterly, forthcoming. [full text]
- Stix, G. (2008): When Markets Beat the Polls, Scientific American Magazine, March 2008. [Abstract]

Markets vs. unaided experts and groups

- Pennock, D. M., Lawrence, S., Giles, C.L. & Nielsen, F.A. (2000). The Power of Play: Efficiency and Forecast Accuracy in Web Market Games, Technical Report 2000-168, NEC Research Institute. [full text]
- For predicting Oscar Award winners, Pennock et al. (2000) compared prices of the Hollywood Stock exchange to expert judgments of five movie columnists. On the day the experts revealed their forecasts, only one of them was better than the market predictions. From the day after, the market outperformed all experts as well as the expert consensus.
- Servan-Schreiber, E. J., Wolfers, J., Pennock, D. M. & Galebach, B. (2004). Prediction Markets: Does Money Matter? Electronic Markets, 14, 243-251. [full text]
- For predicting the results of NFL games, Servan-Schreiber et al. (2004) compared the forecasts of two markets to those of 1,947 self-selected individuals. At the end of the season, the markets ranked 6th and 8th compared to the individuals. The human average – which would be the outcome of a classical survey – ranked 39th.

Markets vs. other forecasting methods

- Chen, K. Y., Plott, C. R. (2002). Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, Social Science Working Paper No.1131, California Institute of Technology, Pasadena. [full text]
- For forecasting sales figures, Chen and Plott (2002) reported on an internal market at Hewlett-Packard that beat the official forecasts of the company in 6 out of 8 events.
- Jones Jr., R. J. (2008). The state of presidential election forecasting – The 2004 experience, International Journal of Forecasting, 24, 308-319. [Abstract]
- Jones (2008) analyzed the forecasts of IEM’s vote-share market for the 2004 election and compared them to traditional polls, a Delphi expert survey, regression models and a combination of all four approaches, the Pollyvote. He concludes that in comparison with most methods of forecasting the popular vote, the IEM was the superior performer.Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]
- Spann and Skiera (2003) compared forecast accuracy of an internal market at a large German mobile phone operator. They found that the market forecasts outperformed were more accurate than four extrapolation models (arithmetic mean, geometric mean, linear trend and exponential trend).

Corporate Markets

- Chen, K.-Y. & Plott, C. R. (2002). Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem. Social Science Working Paper No.1131, California Institute of Technology, Pasadena. [Full text]
- Cowgill, B., Wolfers, J. & Zitzewitz, E. (2008). Using prediction markets to Track Information Flows: Evidence from Google, working paper. [Full text]
- Ortner, G. (1997). Forecasting Markets – An Industrial Application: Part I, working paper, TU Vienna. [Full text]
- Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]

Decision Markets

- Hanson, R. (1999). Decision Markets, IEEE Intelligent Systems, 14, 16-19.

Manipulation

- [Except] Hansen et al. (1998), most empirical studies report that manipulative attacks on result accuracy have not been successful historically (Rhode and Strumpf 2006), in the laboratory (Hanson et al. 2006), and in the field (Camerer 1998).
- Camerer, C. (1998): Can Asset Markets Be Manipulated? A Field Experiment with Racetrack Betting, Journal of Political Economy, 106(3), 457-482. [Abstract]
- Hansen, J., Schmidt, C. & Strobel, M. (2004). Manipulation in Political Stock Markets – Preconditions and Evidence, Applied Economics Letters, 11, 459-463. [Abstract]
- Hanson, R., Oprea, R. & Porter, D. (2006). Information Aggregation and Manipulation in an Experimental Market, Journal of Economic Behavior & Organization, 60, 449-459. [full text]
- Rhode, P. W., and Strumpf, K. S. (2006). Manipulating Political Stock Markets: A Field Experiment and a Century of Observational Data, Working Paper, University of North Carolina(2006). [full text]

-

More research papers on prediction markets

-

(with thanks to Chris Hibbert of Zocalo)

-

-
Iowa Electronic Markets – (IEM) – (Henry B. Tippie College of Business, University of Iowa, Iowa, U.S.A.)

Interdisciplinary Center for Economic Science – (ICES) – (George Mason University, Virginia, U.S.A.)

Economic Science Institute – (ESI) – (Chapman University, California, U.S.A.)

HP Labs – (Hewlett-Packard, Palo Alto, California, U.S.A.)

Laboratory for Experimental Economics and Political Science – (EEPS) – (CalTech, California, U.S.A.)

Vecon Lab – (University of Virginia, Virginia, U.S.A.)

MIT Center for Collective Intelligence – (CCI) – (Cambridge, Massachusetts, U.S.A.)

Yahoo! Research Labs – (Yahoo!, Pasadena, California, U.S.A.)

Laboratory for Economics Management and Auctions – (LEMA) – (Penn State University, Pennsylvania, U.S.A.)

MicroSoft Research – (MicroSoft, Redmont, Washington, U.S.A.)

Google‘s Economics Group – (Google, Mountain View, California, U.S.A.)

Center for Prediction Markets – (National Chengchi University, Taiwan)

Knowledge Lab – (Witten/Herdecke U., Cologne, Germany, E.U.)

-

Contact me to submit other entries.

-

WARNING: Even though the Intel director uses 15 times the term “prediction markets” in this paper, the forecasting tool they have been using is another form of information aggregation mechanism.

—

Via the absolutely indispensable but nevertheless extremely modest George Tziralis, this article in the Intel Technology Journal of May 2007:

The Spectrum of Risk Management in a Technology Company – Using Forecasting Markets to Manage Demand Risk – (PDF) – by Intel Corporation‘s Jay W. Hopman – 2007-05-16

—

- Abstract

Intel completed a study of several generations of products to learn how product forecasts and plans are managed, how demand risks manifest themselves, and how business processes contend with, and sometimes contribute to, demand risk. The study identified one critical area prone to breakdown: the aggregation of market insight from customers. Information collected from customers and then rolled up through sales, marketing, and business planning teams is often biased, and it can lead to inaccurate forecasts, as evidenced by historical results. A research effort launched in 2005 sought to introduce new methodologies that might help crack the bias in demand signals. We worked with our academic partners to develop a new application, a form [???] of prediction market, integrated with Intel’s regular short-term forecasting processes. The process enables product and market experts to dynamically negotiate product forecasts in an environment offering anonymity and performance-based incentives. To the extent these conditions curb bias and motivate improved performance, the system should alleviate demand miscalls that have resulted in inventory surpluses or shortages in the past. Results of early experiments suggest that market-developed forecasts are meeting or beating traditional forecasts in terms of increased accuracy and decreased volatility, while responding well to demand shifts. In addition, the new process is training Intel’s experts to improve their use and interpretation of information.

- Introduction

[...] Tackling demand risk and other challenges requires moving information around decentralized organizations in new ways. If employees across Intel’s many functional groups have information and insights that can help inform our planning and forecasting decisions, we need a way to aggregate that information and turn it into intelligence. Prediction markets are a potential solution to this problem and have been written about extensively for the past five to ten years. Our research discovered that, despite the buzz around prediction markets, the integration of prediction markets and similar Information Aggregation Mechanisms (IAMs) into organizational forecasting processes is still in its infancy. Popular stories on prediction markets still frame the potential as being greater than the demonstrated value, and reports of usage at companies such as Hewlett Packard, Microsoft, Google, Eli Lilly, and others suggest that application is often viewed as experimental and that markets are largely separate from other organizational forecasting processes.

- Challenges to Anticipating Market Demand

[...] Decentralized organizations must find a means of transmitting business context; in other words, instead of transmitting mere data sets, they must transmit information and intelligence from employees who have it to employees who need it to make decisions and plans. We learned that Intel has many informal networks that attempt to move that knowledge across the organization, but these networks have many failure modes: turnover of employees in key positions, limited bandwidth of each individual and team, and difficulty systematically discovering the important information to be learned (stated differently, whom to include in the network). [...]

- Market Mechanisms as Forecasting Tools

[...] In our research at Intel we are extending the idea of prediction markets to create “forecasting markets,” which are essentially prediction markets or similar IAMs integrated into the company’s standard, ongoing forecasting processes. Participants reveal not just an expected outcome but a series of expected outcomes [???] for the same variable over time. So, the forecasting market captures individual and collective assessments about trends such as increasing or decreasing demand just as weather forecasts anticipate warming and cooling trends. [...] Anonymity helps prevent biases created by the presence of formal or informal power, the social norms of group interaction, and expectations of management. [...]

- Design Considerations and Elections

[...] Our overall design structures each investment as a decision based on both the individual’s expectations for the outcome and the aggregate group prediction. Participants weigh owning lower percentages of more likely outcomes against higher percentages of less likely outcomes. [...]

- Results

We are using three primary measures to assess the performance of our markets: accuracy, stability, and timely response to genuine demand shifts. Having run pilot markets for approximately 18 months, we are starting to get a sense for how the markets are performing. Although the market forecasts and official company forecasts are not independent, it is nonetheless interesting to compare the signals and then assess how effectively they are working together. In terms of accuracy, the markets are producing forecasts at least the equal of the official figures and as much as 20% better (20% less error), an impressive result given that the official forecasts have set a rather high standard during this time period with errors of only a few percent. In the longest sample to date, six of eight market forecasts fell within 2.7% of actual sales. The accuracy of the official and market forecasts has been remarkably good, well within the stated goal of +/- 5% error for all but a few individual monthly forecasts. [...] We are also amused that although we never publish the list of participants and winners, everyone knows who participated and who won. [...]

- Challenges

[...] As we propose market mechanisms to aid with forecasting, potential participants and managers have most often expressed three concerns: incentives, anonymity, and groupthink. [...]

- Summary and Conclusions

[...] The key drivers that we believe have led to strong performance are 1) anonymity and incentives, which encourage honest, unbiased information, 2) the averaging of multiple opinions, which produces smooth, accurate signals, and 3) feedback, which enables participants to evaluate past performance and learn how to weigh information and produce better forecasts. [...] [Prediction markets] are a new approach toward business management, promising, and at the same time frightening to potential adopters. As with many such innovations, starting small and running in parallel to existing processes are keys to success. As our trials are demonstrating excellent results at remarkably low cost, expanding their use at Intel is a natural and expected outcome.

- Sidebar: Five Categories of Considerations for Designing Information Aggregation Mechanisms

Information – Integration – Inclusion – Interface – Incentives

UPDATE: Robin Hanson has a comment…

It is great to see another comparison, but it would be more persuasive if we could see a bit more detail. How many markets have been run, do they use the last price or an average for their comparisons, was the comparison mechanism able to see the market prices or vice versa, and so on.

UPDATE #2: Deep Throat…

There are not enough details in the paper.

UPDATE #3: Deep Throat #2…

It seems quite light on data and the references are pretty unimpressive.

UPDATE #4: Chris Masse thinks that this paper is significant for two reasons. Number one, it says that internal prediction markets do work at Intel and that they intend to go on. Number two, Intel has integrated its internal prediction markets into their overall business forecasting system. It’s the first that a Fortune-500 firm states that publicly, if I’m correct.

UPDATE #5: Some people in the field of prediction markets think that the Intel mechanism has nothing to do with trading and is closer to a survey mechanism.

UPDATE #6: INTEL BUSINESS CASE: Does Intel really use internal prediction markets?

UPDATE #7: Emile Servan-Schreiber:

[…] It is fairly obvious from reading the INTEL case study that they are not using a trading market at all but rather something closer to HP’s BRAIN. […]

Earnings Per Share Event Derivatives
Wal-Mart Stores
Exxon Mobil
3M
Amazon.com
Apple
Boeing
Caterpillar
Citigroup
Ford Motor
Genentech, Inc.
Google
Halliburton
Intel
McDonald’s
Microsoft
Motorola
Pfizer
Yahoo
Bebe
Costco Wholesale
Dell
Electronic Arts (EA)
Hewlett-Packard
NYSE Euronext
Time Warner
UAL
Walt Disney

Merger And Acquisition Event Derivatives
Siruis/Xm
Yahoo/Microsoft
Newscorp /Dow Jones
Hershey / Cadbury
ISe / Deutsche Bourse
ISE / NYSE
Nasdaq / PHLX
Google / Salesforce
Upperdeck / Topps

Quite impressive list.

Previous: EPS Prediction Markets = Earnings Per Share Prediction Markets

Professor Robin Hanson asserts that, overall, prediction markets done well compared with every other traditional forecast and individual expert. [...]

Existing prediction markets in Pharma Prediction markets are beginning to get attention in the pharmaceutical industry.

A couple years ago, Eli Lilly ran an internal prediction market where a dozen sales people traded contracts related to the monthly sales of three drugs. Predicting these three drugs over three months, they made a total of nine forecasts. They found that the prediction market forecasts were more accurate the “official” forecast (obtained by a traditional method based on projecting past sales into the future) six of the nine times. The prediction market in this example was relatively small, and yet the results still beat the best alternative. Perhaps with a larger, more diverse group, an even better forecast would have been obtained.

Not to be outdone, Pfizer will start using a small scale prediction market model developed for Hewlett Packard’s internal forecasts this year.

It also seems markets with trading doctors have produced accurate data on disease patterns. These have practical implications on the prescription rate of certain drugs. A recent experiment in Iowa, published by The Infectious Diseases Society of America, concluded that:

“Prediction markets… may be useful for tracking and forecasting emerging infectious diseases, such as severe acute respiratory syndrome and avian influenza, by aggregating expert opinion quickly, accurately, and inexpensively. Data from a pilot study in the state of Iowa suggest that these markets can accurately predict statewide seasonal influenza activity 24 weeks in advance… Information revealed by prediction markets may help to inform treatment, prevention, and policy decisions.”

Despite all this, a recent survey of pharma forecasting experts showed that only 13% had heard of prediction markets applied to the Pharma industry. So how should this industry be looking to use prediction markets?

No doubt individual pharma companies will start to and continue to use prediction markets. But imagine a prediction market that brought together the expertise of a wide range of pharma experts from different companies- experts from diverse fields, with a huge variety of expertise. Based on what we have seen so far, the results from such a market could provide some very valuable forecast data. At Eyeforpharma, bringing together experts from across the industry is what we do. This is why we think we may be able to make this a reality.

PharmaDAQ

Credit: Declan McCullagh/CNET News.com:

Robin Hanson, an economics professor at George Mason University and an early proponent of using prediction markets to judge government policies, stands at the podium. On the left is Eric Zitzewitz, an economics professor at Stanford University’s business school.

At a micro-conference held at Yahoo’s headquarters in Sunnyvale, Calif., representatives from Google, Yahoo, Microsoft and Hewlett-Packard described their experiments with prediction markets.

- CNET: Tech lessons learned from the wisdom of crowds

- Midas Oracle .ORG / Chris Masse: Prediction Markets: Tapping the Wisdom of Crowds + Yahoo!’s new currency: Yootles + Yootopia project

—

DOCTOR DAVID PENNOCK (OF YAHOO! RESEARCH),

WOULD IT BE POSSIBLE TO SEE A REPLAY OR TO DOWNLOAD THE MEDIA FILES? YAHOO! PLAYLIST IS MUTE, NOW.

THANKS. APPRECIATED.

—

- The replay of the David Pennock-endorsed event: 100k | 300k