Midas Oracle .ORG

Jed Christiansen July 23rd, 2008

I’ve been running a niche prediction market site for research over the last three summers. Recently, some of these play-money markets overlapped with real-money markets currently on-going on Betfair.

In my post over at Mercury’s Blog, I discuss how I’ve used play-money market wisdom to take advantage of some poor market-makers on the real-money Betfair markets. Specifically, the favourite on one particular play-money market has a probability to win of ~80% (which I think is fairly accurate). I managed to buy that same contract on Betfair at the equivalent of a 40% probability. Similar examples still exist because of market makers that skewed initial odds towards long-shots (at least by play-money market standards).

Perhaps this will strike some conversation on potential arbitrage between play-money and real-money prediction markets.

Tags: arbitrage, event derivative markets, play-money prediction markets, prediction markets, real-money prediction markets

• All Guest Authors's Posts , Analysis (Data) , Arbitrages , Exchanges & Markets , Market Efficiency , Prices & Probabilities , Trading
• Comments(0)

Chris F. Masse June 11th, 2008

As seen by Andreas Graefe…

IIF’s SIG on Prediction Markets

Research Papers

Basics

Several studies explain the concept of prediction markets and provide useful summaries of the method, e.g.

- Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]
- Wolfers, J. & Zitzewitz, E. (2006). Prediction Markets in Theory and Practice, New Palgrave Dictionary of Economics and the Law (in press). [Full text]
- Wolfers, J. & Zitzewitz, E. (2004). Prediction Markets, Journal of Economic Perspectives, 18, 107-126. [Full text]
- An overview and classification of 152 studies on prediction markets, published between 1991 and 2006, is provided by
Tziralis, G. & Tatsiopoulos (2007). Prediction Markets: An Extended Literature Review, Journal of Prediction Markets, 1, 75-91. [Full text]

Evidence on the accuracy of prediction markets

This section summarizes research that analyzes the relative performance of prediction markets and other forecasting methods.

Markets vs. polls (election forecasting)

- Berg, J., Nelson, F. & Rietz, T. (2008). Prediction Market Accuracy in the Long Run, International Journal of Forecasting, 24, 283-298. [full text]
- Erikson R. S. & Wlezien C. (2007). Are Political Markets Really Superior to Polls as Election Predictors? Public Opinion Quarterly, forthcoming. [full text]
- Stix, G. (2008): When Markets Beat the Polls, Scientific American Magazine, March 2008. [Abstract]

Markets vs. unaided experts and groups

- Pennock, D. M., Lawrence, S., Giles, C.L. & Nielsen, F.A. (2000). The Power of Play: Efficiency and Forecast Accuracy in Web Market Games, Technical Report 2000-168, NEC Research Institute. [full text]
- For predicting Oscar Award winners, Pennock et al. (2000) compared prices of the Hollywood Stock exchange to expert judgments of five movie columnists. On the day the experts revealed their forecasts, only one of them was better than the market predictions. From the day after, the market outperformed all experts as well as the expert consensus.
- Servan-Schreiber, E. J., Wolfers, J., Pennock, D. M. & Galebach, B. (2004). Prediction Markets: Does Money Matter? Electronic Markets, 14, 243-251. [full text]
- For predicting the results of NFL games, Servan-Schreiber et al. (2004) compared the forecasts of two markets to those of 1,947 self-selected individuals. At the end of the season, the markets ranked 6th and 8th compared to the individuals. The human average – which would be the outcome of a classical survey – ranked 39th.

Markets vs. other forecasting methods

- Chen, K. Y., Plott, C. R. (2002). Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, Social Science Working Paper No.1131, California Institute of Technology, Pasadena. [full text]
- For forecasting sales figures, Chen and Plott (2002) reported on an internal market at Hewlett-Packard that beat the official forecasts of the company in 6 out of 8 events.
- Jones Jr., R. J. (2008). The state of presidential election forecasting - The 2004 experience, International Journal of Forecasting, 24, 308-319. [Abstract]
- Jones (2008) analyzed the forecasts of IEM’s vote-share market for the 2004 election and compared them to traditional polls, a Delphi expert survey, regression models and a combination of all four approaches, the Pollyvote. He concludes that in comparison with most methods of forecasting the popular vote, the IEM was the superior performer.Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]
- Spann and Skiera (2003) compared forecast accuracy of an internal market at a large German mobile phone operator. They found that the market forecasts outperformed were more accurate than four extrapolation models (arithmetic mean, geometric mean, linear trend and exponential trend).

Corporate Markets

- Chen, K.-Y. & Plott, C. R. (2002). Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem. Social Science Working Paper No.1131, California Institute of Technology, Pasadena. [Full text]
- Cowgill, B., Wolfers, J. & Zitzewitz, E. (2008). Using prediction markets to Track Information Flows: Evidence from Google, working paper. [Full text]
- Ortner, G. (1997). Forecasting Markets - An Industrial Application: Part I, working paper, TU Vienna. [Full text]
- Spann, M. & Skiera, B. (2003). Internet-based Virtual Stock Markets for Business Forecasting, Management Science, 49, 1310-1326. [Full text]

Decision Markets

- Hanson, R. (1999). Decision Markets, IEEE Intelligent Systems, 14, 16-19.

Manipulation

- [Except] Hansen et al. (1998), most empirical studies report that manipulative attacks on result accuracy have not been successful historically (Rhode and Strumpf 2006), in the laboratory (Hanson et al. 2006), and in the field (Camerer 1998).
- Camerer, C. (1998): Can Asset Markets Be Manipulated? A Field Experiment with Racetrack Betting, Journal of Political Economy, 106(3), 457-482. [Abstract]
- Hansen, J., Schmidt, C. & Strobel, M. (2004). Manipulation in Political Stock Markets - Preconditions and Evidence, Applied Economics Letters, 11, 459-463. [Abstract]
- Hanson, R., Oprea, R. & Porter, D. (2006). Information Aggregation and Manipulation in an Experimental Market, Journal of Economic Behavior & Organization, 60, 449-459. [full text]
- Rhode, P. W., and Strumpf, K. S. (2006). Manipulating Political Stock Markets: A Field Experiment and a Century of Observational Data, Working Paper, University of North Carolina(2006). [full text]

-

More research papers on prediction markets

-

Tags: academic papers, academic research, betting exchange, betting markets, event derivative markets, prediction exchanges, prediction markets, research

• Accuracy (Absolute / Relative) & Precision , All Best Posts Ever , Analysis (Meta) , Economics , Exchanges & Markets , Forecasting , Insider Trading , Manipulations , Market Efficiency , Resources & References
• Comments(0)

Chris F. Masse May 23rd, 2008

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

Via Daniel Horowitz (Business and Technology Consultant)

Blip.TV — (FLV file)

-

-

-

It’s cool.

-

-

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.

-

DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

-

PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]

We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.

The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]

Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]

The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]

Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.

Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]

Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]

In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]

-

FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.

-

APPENDIX:

-

Bo Cowgill’s precision point on micro-geography:

Below you can see a snapshot of trading in one of our offices. The areas where employees are making profitable decisions is green, and the areas where employees are making unprofitable decisions is red. There are about 16 profitable traders in that big green blotch in the middle!

-

More information from our previous blog post on the Google paper

-

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

Tags: Bo Google, conference, corporate prediction markets, enterprise prediction markets, event derivative markets, Google, Google's enterprise prediction markets, Google's prediction exchange, Google's prediction markets, intenal prediction markets, internal prediction markets, Money-Tech, O'Reilly, prediction markets, private prediction markets, video

• Accuracy (Absolute / Relative) & Precision , Analysis (Meta) , Business Cases & Case Studies , Exchanges & Markets , Market Efficiency
• Comments(1)

Chris F. Masse May 1st, 2008

Zubin Jelveh

-

Tags: betting markets, Economics, inefficiencies, Justin Wolfers, Sports, spread betting

• Analysis (Data) , Betting , Economics , Market Efficiency
• Comments(0)

Chris F. Masse March 26th, 2008

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

VIDEO — Bo Cowgill on Google’s enterprise prediction markets — O’Reilly Money:Tech

Blip.TV — (FLV file)

-

-

-

-

I’ll first publish some excerpts from the paper on Google’s enterprise prediction markets (set up by Bo Cowgill, Doug Banks, Patri Friedman, Ilya Kirnos, Piaw Na and Hal Varian), and then I’ll list all the interesting discussion points that this paper has generated on the Web…

-

Here’s Bo Cowgill’s face pic, for those who don’t know him yet:

Bo Cowgill - LinkedIn profile

-

-

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.

-

DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

-

PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]

We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.

The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]

Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]

The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]

Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.

Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]

Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]

In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]

-

FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.

-

APPENDIX:

-

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

Via Zubin Jelveh, Bo Cowgill’s precision point on micro-geography:

Below you can see a snapshot of trading in one of our offices. The areas where employees are making profitable decisions is green, and the areas where employees are making unprofitable decisions is red. There are about 16 profitable traders in that big green blotch in the middle!

-

Eric Zitzewitz in The New York Times:

Peoples’ primary job isn’t to trade these commodities. What we are picking up is communication on ‘low priority topics.’ But that’s how creative ideas come about.

-

Bo Cowgill in Network World:

If you let people bet on things anonymously, they will tell you what they really believe because they have money at stake. This is a conversation that’s happening without politics. Nobody knows who each other is, and nobody has any incentive to kiss up. [...]

As a philosophical matter, Google likes to pack people in tight… so they can share information. As a company gets larger, people don’t always adhere to the founding tenets, or if they’re in a big building they’ll spread out because it’s more comfortable. This was something where we could say ‘there’s value and we can measure that, and we can compare it to what it’s like when people e-mail.’

-

Bo Cowgill (answering Mat Fogarty’s remark):

[...] I don’t think that short-aversion explains the optimistic bias revealed in the Google markets. Because of the IEM-like structure of the markets, no shorting was necessary to make a pessimistic bet. If you wanted to bet pessimistically, you could simply buy the pessimistic outcome. In other words: The optimistic bias was not only saw an aversion to shorting optimistic outcomes. It was also an aversion to buying pessimistic ones. We did observe a general aversion to shorting, which is bad for the market’s efficiency and accuracy. Better terminology could overcome this. More on these two biases in a forthcoming post.

-

The 12 principles that guide programming at Google:

1. All developers work out of a ~single source depot; shared infrastructure!
2. A developer can fix bugs anywhere in the source tree.
3. Building a product takes 3 commands (”get, config, make”)
4. Uniform coding style guidelines across company
5. Code reviews mandatory for all checkins
6. Pervasive unit testing, written by developers
7. Unit tests run continuously, email sent on failure
8. Powerful tools, shared company-wide
9. Rapid project cycles; developers change projects often; 20% time
10. Peer-driven review process; flat management structure
11. Transparency into projects, code, process, ideas, etc.
12. Dozens of offices around world => hire best people regardless of location

The Google paper made a social graph of these code reviews and compared trading habits among circle of co-reviewers. See Table #2 and Table #3.

-

Economist Michael Giberson has more on “Prediction markets and the flow of information inside organizations“.

[...] With private, anonymous trading, the choice to disclose information via the market dramatically revises the mental decision calculus involved. The trader is rewarded if right and penalized if wrong, but in either case the disclosure and net reward is a private matter rather than social event. (At least until you brag about it around the water cooler.) [C]andor should be rewarded and incentives designed to encourage it. Prediction markets provide incentives for candor. Not only that, but over time the traders with useful candor are encouraged by accumulated gains, while blowhards find their accounts diminished. It is true that prediction market prices present relatively limited signals. Prices may go up or down, but they never say why. But with a signal, at least someone knows to start asking “why” and that is better than not knowing.

-

Concurring with Justin Wolfers, here’s Tim Harford on the flow of information around Google’s physical office spaces:

[...] The results were striking. Clear correlations existed between the trading behaviour of certain groups of employees. But they were not explained by shared interests or by social connections. Having the same immediate boss only explains a little about information flows. No, it is the office layout that matters: people who sit near each other tend to know the same things, as evidenced by making similar trades on the prediction markets. Social and professional proximity matters very little for the flow of information: physical proximity is almost everything. Specialists in organisational behaviour have known for a while that people tend to interact much more with those who sit nearby, but it has never been clear whether that was just social grooming. Now we know that real information is flowing. We keep being told that because of cheap, ubiquitous communication technology, distance is dead. But if there was ever a company that we should expect to exemplify that idea, surely it was Google. This research suggests that it is as important as ever to be sitting in the right place.

-

Reading Tim Harford, an objection popped up in my mind. If we had ben following this micro-geography model all along, then we would never have had open-source software (e.g., Linux, Mozilla FireFox, WordPress) and open knowledge (e.g., Wikipedia, group blog BoingBoing). In all these instances, collaboration has been done over the Internet by people who don’t know each other in person —although they happen to meet, once in a while . And yet these open projects have contributed greatly, in my view, to the making of our global digital civilization.

-

Enterprise prediction market consultant Jed Christiansen:

I’ve seen this in other markets I’ve run, as well. At the beginning traders are all over the place, with a fair bit of volatility and inaccuracy. But pretty quickly this settles out. I believe it’s a combination of “dumb” traders getting frustrated and stopping, and all traders becoming more sensitive to the risks they’re taking. [...]

Google used a CDA model for their markets. This created arbitrage opportunities, when the sum of the bid prices was more than $1, and when the sum of the ask prices was less than $1. The authors found 1,747 instances of the former, and 495 instances of the latter. As they noted, this demonstrates an aversion to short selling contracts. What was interesting to me was how the market reacted to this. The median duration of any of these arbitrage opportunities was just two minutes, demonstrating that they were correctable. Even better is this: “One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle.” All it takes is one person and a little programming knowledge to make the market much more efficient and liquid! I would never discourage this kind of behaviour. [...]

Bo Cowgill’s reacting about the organizational-behavior argument:

[...] This is not the first reaction along these lines. I am perplexed by the response. I can understand why other companies may not want to replicate our analysis of information flows. Perhaps it wouldn’t be worth the effort. Perhaps they would get identical results. And perhaps the company wouldn’t have the all the necessary data.

However, I expected that people could easily see value in the analysis of granular trade-by-trade data — especially if that data is joined with data about traders and outside events happening at the moment of the trades. We described one very generic application of this approach, but you can imagine much more actionable and company-specific ones.

I will mention one: The data contains real-time metrics on the distribution of knowledge and attitudes within a firm at a highly granular level. You can get metrics on for specific of the firm, for specific classes of employees and for specific topics. You can do this for either customers or employees, and have the metrics for any moment in time. The quality of these metrics will be extremely strong, because participants have been incentivized to reveal their true expectations.

Our analysis spoke in very general terms about the flow of information between Google employees — we don’t reference specific groups or draw distinctions between them — which is where a lot of actionable data was. Trade-by-trade data can reveal characteristics of specific working groups: What they know, how they feel, how they process and share information and how all of that changes over time. I didn’t try to put any of this in the paper because the conclusions would be sensitive, and I thought this application was pretty obvious to anybody who understood our methodology.

UPDATE: Our findings about the clustering of attitudes should also inform anyone who thinks that diversity is important for crowd-wisdom applications — as James Surowiecki famously suggests in The Wisdom of Crowds.

Our analysis suggests that groupthink primarily happens within language networks and small physical spaces (with social/professional networks playing a secondary role, and demographic networks playing a non-existent one). Remember that as you’re selecting your traders. If they already work/sit/chat together, the groupthink may already exist and the market won’t cure it.

Reacting on Jed Christiansen’s other post, Bo Cowgill adds:

I’ve also heard that other companies would find it impossible to analyze the interaction between their market and the organization. Why? Lack of data. Our analysis benefited from a wealth of internal data (including GPS coordinates of offices) that other companies don’t store.

You may be surprised at how much data average companies really have. For example, Google had social network surveys; many companies do not. However, many standard corporate applications (such as email, calendars, telephones and code reviews) contain implicit social networks that can be used in place of data gathered from surveys.

Or, consider this: I recently met with people from Google’s real estate management group. Turns out, they have records of the floorplans of Google’s offices in electronic format. Not only can someone use these records to find the distance between offices (without GPS coordinates) — you can also find the total area and perimeter of each office, which desks are open (cube-style) vs. enclosed, the walking distances between offices and more.

Surprised and impressed, I asked if it was typical for companies to have all of this information. The response was: “Any Fortune 1000 company would have this data about their offices.” Everyone in the room said his previous employer had the same data — typically managed through computer-aided facility management systems such as Archibus or Infor.

Bo Cowgill:

Some more remarks about applications that combine prediction markets and organizational data (org charts, social networks, seating locations). The obstacle to these applications is not a lack of data. Jed mentions privacy concerns — and if he thinks this is a big obstacle then I’d be interested in discussing his thoughts.

A bigger problem is that that current prediction market vendors and consultants cannot support these applications. At heart, these vendors are software engineers and salespeople at heart, not statisticians or data miners. They want to write one system that can support lots of clients. At conferences, one hears PM vendors complain about having to do “customization” work for clients.

This approach would not work for the applications I describe for two reasons:

1. The inputs for different clients won’t be the same. Each client’s organizational data will likely take a different structure. This makes it difficult for prediction market vendors to architect a single system that can served many clients (yet another challenge with integrating markets with other corporate IT services).
2. The outputs for different clients won’t be the same. The business relevance and statistical power of each analysis will differ with each client’s data.

Prediction market vendors may also need to familiarize themselves with the statistical learning methods necessary to fully utilize these rich datasets. So what’s the solution? First, move to a software-and-consulting model. By ‘consulting,’ I don’t mean ‘consulting on how to implement the market.’ I’m talking about helping the client solve its problem using a variety of data, including prediction market data.

Second, the vendors also need to pitch prediction markets as more than a forecasting tool. People in the business world commonly identify as data junkies — probably more so than they identify with the ‘wisdom of crowds’ ethos. It is unclear how much companies really care about accurate forecasting anyway.

-

Robin Hanson:

Info Value = the added accuracy the markets provide relative to other mechanisms, times the value that accuracy can give in improved decisions, minus the cost of maintaining the markets, relative to the cost of other mechanisms.

A highly accurate market has little value if other mechanisms can provide similar accuracy at a lower cost, or if few substantial decisions are influenced by accurate forecasts on its topic.

-

Bo Cowgill:

Although PM experiments make a company look ‘cool,’ disclosing specific high profile applications isn’t necessarily in the firm’s interests. I can vouch for this based on my own experience. Of a firm’s major stakeholders, we don’t know who would find a particular application unfair or irresponsible. This is particularly true if the application will create winners and losers — as would often be the case with high value markets.

Robin Hanson:

[...] I meant trying to field the highest value applications. That is naturally measured in accounting terms - value minus cost. Measures of popularity or familiarity would not at all be the same thing.

Bo Cowgill:

[...] I did not say anything about the popularity of the markets. I proposed measuring the number of employees working in a job that has a prediction market on it. This is a function of the firms decision to implement markets on a wide variety of relevant topics for the business.

I know this is a heuristic, and there would be greater value in trying to measure the value added in dollars. However, a monetization study is likely to be 1) more time consuming to produce, and 2) just as unreliable.

We aren’t likely to get honest answers from people by asking them to estimate the value of additional information at various specific hypothetical moments in the past. People are not good at making these type of estimations, especially when there is no incentive to get it right (and a lot of reasons to get it wrong).

On the costs side: Even if we had good data about how much individual employees were spending on the site browsing, etc — it is unclear how to price such their time. Is it work, or is it leisure? Should we model this as employees having an hourly rate (even though most are salaried)? I know you had some thoughts on this before which I don’t remember (do feel free to share), but I’m not convinced these issues can be resolved in a persuasive way.

Once people realize what’s going on with the methodology of such research, they’ll realize what a totally hackable and unreliable study it is — and it will lose its persuasive value at Google as well as externally.

In summary: Doing a value-of-information calculation on prediction markets itself does not seem to offer very much information value. Because of the low rigor, it would offer little additional persuasive value at a great cost.

Robin Hanson:

I can’t have much optimism about a business practice whose proponents aren’t even willing to try to offer a cost-benefit calculation. You could count how many employees had ever gone to a TQM meeting, but that wouldn’t tell you if TQM is valuable or not.

[Here's Robin Hanson's website. For your information (if you are a newbie), Robin Hanson is the most advanced researcher in the field of prediction markets. He co-invented the modern-day prediction markets, the concept of decision markets, and a new marked design, the Market Scoring Rule.]

-

Voila. Thanks for your attention. Feel free to publish a comment, just below… or blog about it —and I’ll update this post with some of your thoughts.

-

Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06

-

Tags: accuracy, Bo Cowgill, corporate prediction markets, efficieny, enterprise prediction markets, Eric Zitwewitz, event derivative markets, Google, Google's enterprise prediction markets, Google's prediction exchange, Google's prediction markets, internal prediction markets, Justin Wolfers, prediction markets, private prediction markets, Robin Hanson

• Accuracy (Absolute / Relative) & Precision , All Best Posts Ever , Analysis (Meta) , Business , Business Cases & Case Studies , Exchanges & Markets , Market Efficiency
• Comments(13)

Chris F. Masse February 25th, 2008

Roger Ehrenberg:

Prediction markets and elections, run by Eric Zitzewitz of Dartmouth. Eric ran though a slew of data looking at sports and financial prediction markets and compared it to election prediction markets. Bottom line, sports and financial prediction markets are highly efficient and closely track actual outcomes, while election prediction markets tend to systematically underprice favorites. This is the way it is in the US. In the UK, where election prediction markets have existed for over a century, they are much more efficient. Two explanations for the US election markets’ relative inefficiency are low repetition vs. high repetition domains (sports and financials have large numbers of contests while elections have far fewer contexts), and that bettors in the US have far less collective experience betting on elections than their UK counterparts. Interesting stuff.

Greg Linden’s post (from which I lifted the sentence posted in the title above)

Inkling Markets’ Adam Siegel

Collective Intelligence Foo Camp

List of attendees.

Prediction markets came up as topic of interest #1 —at least online.

Wiki participation was low, though.

I was invited, but couldn’t make it, due to time pressure. From the e-mail they [*] sent me at the time:

CI Foo is, by design, far from a typical conference. The participants are the presenters–so be prepared to demo or speak. We’ll schedule some sessions ahead of time, and set up a process for participants to announce sessions they want to offer. Expect the kind of exchange that happens at the best conferences during breaks and late into the night, plus time for fun at Google’s legendary campus [the "GooglePlex"]. The result will be an informal but intense and eye-opening couple of days.

[*]

1. Gary Flake, Microsoft Live Labs;
2. Tom Malone, Director, MIT Center for Collective Intelligence;
3. Tim O’Reilly, O’Reilly Media;
4. Hal Varian, Google Chief Economist.

Tags: CI Foo, Collective Intelligence Foo Camp, Google, GooglePlex, O'Reilly

• Accuracy (Absolute / Relative) & Precision , Analysis (Meta) , Events & Meetings , Exchanges & Markets , Market Efficiency , Wisdom of Crowds & Collective Intelligence
• Comments(0)

Nigel Eccles December 20th, 2007

In financial markets there is strong evidence to suggest that news gets priced into markets within 15 minutes of its release and sometimes even more quickly. Recent research into prediction markets suggests that they aren’t nearly as efficient with researchers from University of Pennsylvania showing that prices on IEM can be predicted using public news flow.

Doing a simple analysis of some the key events in the 2008 Presidential Elections against prices on Intrade shows that on discrete events there is a clear relationship between prices and news flow. However over longer periods the relationship is not always clear.

On the 4th of March CBS announced the results of a straw poll conducted at the conservative PAC convention in Washington DC. They picked Romney as their favourite. Romney’s price on Intrade lifted immediately where it stayed for about a week.

On the 11th of April the Fred Thompson revealed on Fox News and ABC Radio that he had been diagnosed with non-Hodgkin’s lymphoma nearly three years prior. The New York Times and other publications picked up the story the next day. Looking at his price chart shows he opened on the 12th of April at 19 but then closed at 15. The next day he opened at 11.2 but then closed at 17, as the story died down.

In both these cases, the news stories the media considered to be the important ones correspond with the news flow that traders thought was important.

However, the most interesting market movement of the year must be the Obama August slide. On the first of August Obama opened on Intrade at 35.5 but by the 24th of that month he had slide to 17.2. He continued sliding hitting a rock bottom of 10.7 on the 14th of October.

The question is what was the news flow on Obama from the 1st of August to the 24th of August? Analysing the news articles in the New York Times suggests a disconnect between what was reported and how the market was reacting. Obama started August badly with a bungled comment on use of nuclear weapons.

Additionally, his continued line that stabilisation of Iraq had been a ‘complete failure’ may also have cost him some points.

However in sum these news items don’t seem to correlate with an 18 point slide. This could lead us to two possible conclusions:

1. The New York Times didn’t report the most market sensitive news affecting Obama in August
2. Obama was over-sold in August and his price did not reflect his true value

Cross-posted from the Hubdub blog.

Tags: political prediction markets, Politics, Predictions, US President, US presidential elections

• All Guest Authors's Posts , Analysis (Data) , Exchanges & Markets , Market Efficiency , Prices & Probabilities
• Comments(3)

Chris F. Masse December 8th, 2007

Panos Ipeirotison commenting on Niall O’Connor’s post:

My understanding is that Betfair odds moved from 1.44 to 1.50 (according to the screenshot in the original posting). While indeed this corresponds to a drop from 69% to 66% (an almost 4% drop in share price), this is not as drastic as a drop from 69% to 50% within such a short period of time. Plus, the Betfair drop from 69% to 66% is comparable with the drop in Intrade (from 67% to 64%).

Also, I am not sure about the liquidity hypothesis for explaining the inefficiency. An alternative explanation is the following:

Political markets are not stock markets. They reflect the aggregate opinion of the traders about public’s intention for the candidate. Notice that we have two levels of beliefs: one for what traders believe about the public’s intentions, and a second for what the public actually intends to vote for.

Not every member of the voting public reads every piece of information. When the same news are being repeated over and over in the mainstream news outlets, then more voters are influenced. Hence, the longer the news about a candidate stay around, the longer the public gets influenced by the same, stale news and changes intentions. This is correspondingly reflected in the prediction markets, potentially in an efficient manner.

This may indicate that it is not that the markets are not efficient, but that the voting public is not “efficient” (i.e., voters do not incorporate all the available information in their voting decisions.)

We can test this hypothesis by testing the efficiency/predictability of prediction markers vs. the efficiency/predictability of non-political markets.

We will work further with George Tziralis on the topic, and we will keep you posted.

Tags: Market Efficiency

• Analysis (Data) , Analysis (Meta) , Exchanges & Markets , Market Efficiency , Prices & Probabilities
• Comments(1)

Niall O'Connor December 7th, 2007

Gilder and Lerman hypothesised that past/present events can potentially assist in predicting future prices in prediction markets. They empirically revealed that prediction markets are surprisingly predictable, even by purely market-historical techniques.

Taking hold of the baton from Gilder and Lerma, Panos Ipeirotis and George Tziralis developed techniques for extracting news flow signals to see whether they can indeed be utilised to predict the future performance of markets on the InTrade prediction exchange. On the question of whether Hillary Clinton will be the Democratic Presidential Nominee in 2008, they noted;

Our sentiment index (in maroon) is close to 1 when we predict that the market will move higher, and it is close to 0 when we predict that the market will move down. Typically, it works pretty well for predicting long periods of price increases and declines. To put our money where our mouth is, the signal for the last few days shows that Hillary’s market price will edge lower in the next few days/weeks.

Following on from this we looked at the Intrade prediction market and the Betfair markets on whether Hillary Clinton will be the Democratic Presidential Nominee in 2008, as of 10.45 GMT on December 3 2007. Whilst the Intrade market suggested that Clinton’s probability of victory was 67%, the Betfair market gave a reading of 69%.

We returned to the Intrade prediction market and the Betfair market on whether Hillary Clinton will be the Democratic Presidential Nominee in 2008 at 08.45 GMT on December 7 2007.

Whilst the Intrade prediction market had previously suggested that Clinton’s probability of victory was 67%, it was now suggesting that her probability of victory was 64%.

The Betfair market which had given a reading of 69% on Decmber 3 as regards her probability of winning the democratic nomination, was now suggesting that her probability of victory was only 50%.

It is quite clear, that the both sets of markets are responding to stale news, with Intrade significantly lagging behind Betfair, as regards its ability to aggregate all available news flow. Those that had sold Clinton on Betfair at 1.44 on December 3, on the back of Panos Ipeirotis and George Tziralis’ advice, are now sitting on a healthy profit. The claim that prediction markets are innefficient would seem to be gathering momentum…. with the most likely cause being the fact that they are not liquid enough.

http://www.bettingmarket.com/predictionstale.htm

Tags: event derivative markets, Market Efficiency, prediction markets

• All Guest Authors's Posts , Analysis (Data) , Exchanges & Markets , Market Efficiency
• Comments(3)

Chris F. Masse November 16th, 2006

Does Liquidity Affect Securities Market Efficiency? - (PDF - Excerpted at CFM) - [previous title: Does Noise Trading Affect Securities Market Efficiency?] - by Paul Tetlock - 2006-11-XX

The basic idea of the paper is simple. I measure liquidity and expected returns for various securities, and show that the two are linked. In an efficient market, the benchmark is that all securities should have zero expected returns. I find that the illiquid securities markets have (close to) zero expected returns, implying that these markets are efficient. But the liquid securities show certain mispricing patterns. The nature of these patterns suggests that individuals’ probability misperceptions are the cause of the mispricing in liquid securities.

Note: the zero expected returns benchmark is a simplification. It’s based on the assumption that the equilibrium price of risk is negligible, which is a good approximation for most securities on TradeSports –e.g., sports contracts, and most of the short-term financial contracts. Obviously, this assumption would fail in conventional financial markets, where risk premiums may be large.

Previous Blog Posts:

- Paul Tetlock on the inner working of TradeSports-InTrade

- No change: Mispricing is greater in illiquid markets + Justin Wolfers’s comment

- Does Liquidity Affect Securities Market Efficiency?

- Short Odds for Ignorance

- Gambling and a New Approach to Regulating Information Markets

External Link:

- TradeSports forum thread

Parting Shot:

Yeah, it was the Paul Tetlock festival, today.

Tags: