WARNING: Even though the Intel director uses 15 times the term “prediction markets” in this paper, the forecasting tool they have been using is another form of information aggregation mechanism.

—

Via the absolutely indispensable but nevertheless extremely modest George Tziralis, this article in the Intel Technology Journal of May 2007:

The Spectrum of Risk Management in a Technology Company – Using Forecasting Markets to Manage Demand Risk – (PDF) – by Intel Corporation‘s Jay W. Hopman – 2007-05-16

—

- Abstract

Intel completed a study of several generations of products to learn how product forecasts and plans are managed, how demand risks manifest themselves, and how business processes contend with, and sometimes contribute to, demand risk. The study identified one critical area prone to breakdown: the aggregation of market insight from customers. Information collected from customers and then rolled up through sales, marketing, and business planning teams is often biased, and it can lead to inaccurate forecasts, as evidenced by historical results. A research effort launched in 2005 sought to introduce new methodologies that might help crack the bias in demand signals. We worked with our academic partners to develop a new application, a form [???] of prediction market, integrated with Intel’s regular short-term forecasting processes. The process enables product and market experts to dynamically negotiate product forecasts in an environment offering anonymity and performance-based incentives. To the extent these conditions curb bias and motivate improved performance, the system should alleviate demand miscalls that have resulted in inventory surpluses or shortages in the past. Results of early experiments suggest that market-developed forecasts are meeting or beating traditional forecasts in terms of increased accuracy and decreased volatility, while responding well to demand shifts. In addition, the new process is training Intel’s experts to improve their use and interpretation of information.

- Introduction

[...] Tackling demand risk and other challenges requires moving information around decentralized organizations in new ways. If employees across Intel’s many functional groups have information and insights that can help inform our planning and forecasting decisions, we need a way to aggregate that information and turn it into intelligence. Prediction markets are a potential solution to this problem and have been written about extensively for the past five to ten years. Our research discovered that, despite the buzz around prediction markets, the integration of prediction markets and similar Information Aggregation Mechanisms (IAMs) into organizational forecasting processes is still in its infancy. Popular stories on prediction markets still frame the potential as being greater than the demonstrated value, and reports of usage at companies such as Hewlett Packard, Microsoft, Google, Eli Lilly, and others suggest that application is often viewed as experimental and that markets are largely separate from other organizational forecasting processes.

- Challenges to Anticipating Market Demand

[...] Decentralized organizations must find a means of transmitting business context; in other words, instead of transmitting mere data sets, they must transmit information and intelligence from employees who have it to employees who need it to make decisions and plans. We learned that Intel has many informal networks that attempt to move that knowledge across the organization, but these networks have many failure modes: turnover of employees in key positions, limited bandwidth of each individual and team, and difficulty systematically discovering the important information to be learned (stated differently, whom to include in the network). [...]

- Market Mechanisms as Forecasting Tools

[...] In our research at Intel we are extending the idea of prediction markets to create “forecasting markets,” which are essentially prediction markets or similar IAMs integrated into the company’s standard, ongoing forecasting processes. Participants reveal not just an expected outcome but a series of expected outcomes [???] for the same variable over time. So, the forecasting market captures individual and collective assessments about trends such as increasing or decreasing demand just as weather forecasts anticipate warming and cooling trends. [...] Anonymity helps prevent biases created by the presence of formal or informal power, the social norms of group interaction, and expectations of management. [...]

- Design Considerations and Elections

[...] Our overall design structures each investment as a decision based on both the individual’s expectations for the outcome and the aggregate group prediction. Participants weigh owning lower percentages of more likely outcomes against higher percentages of less likely outcomes. [...]

- Results

We are using three primary measures to assess the performance of our markets: accuracy, stability, and timely response to genuine demand shifts. Having run pilot markets for approximately 18 months, we are starting to get a sense for how the markets are performing. Although the market forecasts and official company forecasts are not independent, it is nonetheless interesting to compare the signals and then assess how effectively they are working together. In terms of accuracy, the markets are producing forecasts at least the equal of the official figures and as much as 20% better (20% less error), an impressive result given that the official forecasts have set a rather high standard during this time period with errors of only a few percent. In the longest sample to date, six of eight market forecasts fell within 2.7% of actual sales. The accuracy of the official and market forecasts has been remarkably good, well within the stated goal of +/- 5% error for all but a few individual monthly forecasts. [...] We are also amused that although we never publish the list of participants and winners, everyone knows who participated and who won. [...]

- Challenges

[...] As we propose market mechanisms to aid with forecasting, potential participants and managers have most often expressed three concerns: incentives, anonymity, and groupthink. [...]

- Summary and Conclusions

[...] The key drivers that we believe have led to strong performance are 1) anonymity and incentives, which encourage honest, unbiased information, 2) the averaging of multiple opinions, which produces smooth, accurate signals, and 3) feedback, which enables participants to evaluate past performance and learn how to weigh information and produce better forecasts. [...] [Prediction markets] are a new approach toward business management, promising, and at the same time frightening to potential adopters. As with many such innovations, starting small and running in parallel to existing processes are keys to success. As our trials are demonstrating excellent results at remarkably low cost, expanding their use at Intel is a natural and expected outcome.

- Sidebar: Five Categories of Considerations for Designing Information Aggregation Mechanisms

Information – Integration – Inclusion – Interface – Incentives

UPDATE: Robin Hanson has a comment…

It is great to see another comparison, but it would be more persuasive if we could see a bit more detail. How many markets have been run, do they use the last price or an average for their comparisons, was the comparison mechanism able to see the market prices or vice versa, and so on.

UPDATE #2: Deep Throat…

There are not enough details in the paper.

UPDATE #3: Deep Throat #2…

It seems quite light on data and the references are pretty unimpressive.

UPDATE #4: Chris Masse thinks that this paper is significant for two reasons. Number one, it says that internal prediction markets do work at Intel and that they intend to go on. Number two, Intel has integrated its internal prediction markets into their overall business forecasting system. It’s the first that a Fortune-500 firm states that publicly, if I’m correct.

UPDATE #5: Some people in the field of prediction markets think that the Intel mechanism has nothing to do with trading and is closer to a survey mechanism.

UPDATE #6: INTEL BUSINESS CASE: Does Intel really use internal prediction markets?

UPDATE #7: Emile Servan-Schreiber:

[…] It is fairly obvious from reading the INTEL case study that they are not using a trading market at all but rather something closer to HP’s BRAIN. […]