Robin Hanson – Overcoming Bias -

-

Robin Hanson at LinkedIn

-

Robin Hanson:

Info Value = the added accuracy the markets provide relative to other mechanisms, times the value that accuracy can give in improved decisions, minus the cost of maintaining the markets, relative to the cost of other mechanisms.

A highly accurate market has little value if other mechanisms can provide similar accuracy at a lower cost, or if few substantial decisions are influenced by accurate forecasts on its topic.

-

Robin Hanson:

[...] I meant trying to field the highest value applications. That is naturally measured in accounting terms – value minus cost. Measures of popularity or familiarity would not at all be the same thing.

-

Robin Hanson:

[One should] try to offer a cost-benefit calculation. You could count how many employees had ever gone to a TQM meeting, but that wouldn’t tell you if TQM is valuable or not.

-

[Here's Robin Hanson's website. For your information (if you are a newbie), Robin Hanson is the most advanced researcher in the field of prediction markets. He co-invented the modern-day prediction markets, the concept of decision markets, and a new marked design, the Market Scoring Rule.]

-

Related Info:

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

-

VIDEO — Bo Cowgill on Google’s enterprise prediction markets — O’Reilly Money:Tech

Blip.TV — (FLV file)

-

-

-

-

I’ll first publish some excerpts from the paper on Google’s enterprise prediction markets (set up by Bo Cowgill, Doug Banks, Patri Friedman, Ilya Kirnos, Piaw Na and Hal Varian), and then I’ll list all the interesting discussion points that this paper has generated on the Web…

-

Here’s Bo Cowgill’s face pic, for those who don’t know him yet:

Bo Cowgill – LinkedIn profile

-

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.

-

DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

-

PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]

We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.

The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]

Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]

The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]

Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.

Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]

Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]

In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]

-

FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.

-

APPENDIX:

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

Via Zubin Jelveh, Bo Cowgill’s precision point on micro-geography:

Below you can see a snapshot of trading in one of our offices. The areas where employees are making profitable decisions is green, and the areas where employees are making unprofitable decisions is red. There are about 16 profitable traders in that big green blotch in the middle!

-

Eric Zitzewitz in The New York Times:

Peoples’ primary job isn’t to trade these commodities. What we are picking up is communication on ‘low priority topics.’ But that’s how creative ideas come about.

-

Bo Cowgill in Network World:

If you let people bet on things anonymously, they will tell you what they really believe because they have money at stake. This is a conversation that’s happening without politics. Nobody knows who each other is, and nobody has any incentive to kiss up. [...]

As a philosophical matter, Google likes to pack people in tight… so they can share information. As a company gets larger, people don’t always adhere to the founding tenets, or if they’re in a big building they’ll spread out because it’s more comfortable. This was something where we could say ‘there’s value and we can measure that, and we can compare it to what it’s like when people e-mail.’

-

Bo Cowgill (answering Mat Fogarty’s remark):

[...] I don’t think that short-aversion explains the optimistic bias revealed in the Google markets. Because of the IEM-like structure of the markets, no shorting was necessary to make a pessimistic bet. If you wanted to bet pessimistically, you could simply buy the pessimistic outcome. In other words: The optimistic bias was not only saw an aversion to shorting optimistic outcomes. It was also an aversion to buying pessimistic ones. We did observe a general aversion to shorting, which is bad for the market’s efficiency and accuracy. Better terminology could overcome this. More on these two biases in a forthcoming post.

-

The 12 principles that guide programming at Google:

1. All developers work out of a ~single source depot; shared infrastructure!
2. A developer can fix bugs anywhere in the source tree.
3. Building a product takes 3 commands (“get, config, make”)
4. Uniform coding style guidelines across company
5. Code reviews mandatory for all checkins
6. Pervasive unit testing, written by developers
7. Unit tests run continuously, email sent on failure
8. Powerful tools, shared company-wide
9. Rapid project cycles; developers change projects often; 20% time
10. Peer-driven review process; flat management structure
11. Transparency into projects, code, process, ideas, etc.
12. Dozens of offices around world => hire best people regardless of location

The Google paper made a social graph of these code reviews and compared trading habits among circle of co-reviewers. See Table #2 and Table #3.

-

Economist Michael Giberson has more on “Prediction markets and the flow of information inside organizations“.

[...] With private, anonymous trading, the choice to disclose information via the market dramatically revises the mental decision calculus involved. The trader is rewarded if right and penalized if wrong, but in either case the disclosure and net reward is a private matter rather than social event. (At least until you brag about it around the water cooler.) [C]andor should be rewarded and incentives designed to encourage it. Prediction markets provide incentives for candor. Not only that, but over time the traders with useful candor are encouraged by accumulated gains, while blowhards find their accounts diminished. It is true that prediction market prices present relatively limited signals. Prices may go up or down, but they never say why. But with a signal, at least someone knows to start asking “why” and that is better than not knowing.

-

Concurring with Justin Wolfers, here’s Tim Harford on the flow of information around Google’s physical office spaces:

[...] The results were striking. Clear correlations existed between the trading behaviour of certain groups of employees. But they were not explained by shared interests or by social connections. Having the same immediate boss only explains a little about information flows. No, it is the office layout that matters: people who sit near each other tend to know the same things, as evidenced by making similar trades on the prediction markets. Social and professional proximity matters very little for the flow of information: physical proximity is almost everything. Specialists in organisational behaviour have known for a while that people tend to interact much more with those who sit nearby, but it has never been clear whether that was just social grooming. Now we know that real information is flowing. We keep being told that because of cheap, ubiquitous communication technology, distance is dead. But if there was ever a company that we should expect to exemplify that idea, surely it was Google. This research suggests that it is as important as ever to be sitting in the right place.

-

Reading Tim Harford, an objection popped up in my mind. If we had ben following this micro-geography model all along, then we would never have had open-source software (e.g., Linux, Mozilla FireFox, WordPress) and open knowledge (e.g., Wikipedia, group blog BoingBoing). In all these instances, collaboration has been done over the Internet by people who don’t know each other in person —although they happen to meet, once in a while . And yet these open projects have contributed greatly, in my view, to the making of our global digital civilization.

-

Enterprise prediction market consultant Jed Christiansen:

I’ve seen this in other markets I’ve run, as well. At the beginning traders are all over the place, with a fair bit of volatility and inaccuracy. But pretty quickly this settles out. I believe it’s a combination of “dumb” traders getting frustrated and stopping, and all traders becoming more sensitive to the risks they’re taking. [...]

Google used a CDA model for their markets. This created arbitrage opportunities, when the sum of the bid prices was more than $1, and when the sum of the ask prices was less than $1. The authors found 1,747 instances of the former, and 495 instances of the latter. As they noted, this demonstrates an aversion to short selling contracts. What was interesting to me was how the market reacted to this. The median duration of any of these arbitrage opportunities was just two minutes, demonstrating that they were correctable. Even better is this: “One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle.” All it takes is one person and a little programming knowledge to make the market much more efficient and liquid! I would never discourage this kind of behaviour. [...]

Bo Cowgill’s reacting about the organizational-behavior argument:

[...] This is not the first reaction along these lines. I am perplexed by the response. I can understand why other companies may not want to replicate our analysis of information flows. Perhaps it wouldn’t be worth the effort. Perhaps they would get identical results. And perhaps the company wouldn’t have the all the necessary data.

However, I expected that people could easily see value in the analysis of granular trade-by-trade data — especially if that data is joined with data about traders and outside events happening at the moment of the trades. We described one very generic application of this approach, but you can imagine much more actionable and company-specific ones.

I will mention one: The data contains real-time metrics on the distribution of knowledge and attitudes within a firm at a highly granular level. You can get metrics on for specific of the firm, for specific classes of employees and for specific topics. You can do this for either customers or employees, and have the metrics for any moment in time. The quality of these metrics will be extremely strong, because participants have been incentivized to reveal their true expectations.

Our analysis spoke in very general terms about the flow of information between Google employees — we don’t reference specific groups or draw distinctions between them — which is where a lot of actionable data was. Trade-by-trade data can reveal characteristics of specific working groups: What they know, how they feel, how they process and share information and how all of that changes over time. I didn’t try to put any of this in the paper because the conclusions would be sensitive, and I thought this application was pretty obvious to anybody who understood our methodology.

UPDATE: Our findings about the clustering of attitudes should also inform anyone who thinks that diversity is important for crowd-wisdom applications — as James Surowiecki famously suggests in The Wisdom of Crowds.

Our analysis suggests that groupthink primarily happens within language networks and small physical spaces (with social/professional networks playing a secondary role, and demographic networks playing a non-existent one). Remember that as you’re selecting your traders. If they already work/sit/chat together, the groupthink may already exist and the market won’t cure it.

Reacting on Jed Christiansen’s other post, Bo Cowgill adds:

I’ve also heard that other companies would find it impossible to analyze the interaction between their market and the organization. Why? Lack of data. Our analysis benefited from a wealth of internal data (including GPS coordinates of offices) that other companies don’t store.

You may be surprised at how much data average companies really have. For example, Google had social network surveys; many companies do not. However, many standard corporate applications (such as email, calendars, telephones and code reviews) contain implicit social networks that can be used in place of data gathered from surveys.

Or, consider this: I recently met with people from Google’s real estate management group. Turns out, they have records of the floorplans of Google’s offices in electronic format. Not only can someone use these records to find the distance between offices (without GPS coordinates) — you can also find the total area and perimeter of each office, which desks are open (cube-style) vs. enclosed, the walking distances between offices and more.

Surprised and impressed, I asked if it was typical for companies to have all of this information. The response was: “Any Fortune 1000 company would have this data about their offices.” Everyone in the room said his previous employer had the same data — typically managed through computer-aided facility management systems such as Archibus or Infor.

Bo Cowgill:

Some more remarks about applications that combine prediction markets and organizational data (org charts, social networks, seating locations). The obstacle to these applications is not a lack of data. Jed mentions privacy concerns — and if he thinks this is a big obstacle then I’d be interested in discussing his thoughts.

A bigger problem is that that current prediction market vendors and consultants cannot support these applications. At heart, these vendors are software engineers and salespeople at heart, not statisticians or data miners. They want to write one system that can support lots of clients. At conferences, one hears PM vendors complain about having to do “customization” work for clients.

This approach would not work for the applications I describe for two reasons:

1. The inputs for different clients won’t be the same. Each client’s organizational data will likely take a different structure. This makes it difficult for prediction market vendors to architect a single system that can served many clients (yet another challenge with integrating markets with other corporate IT services).
2. The outputs for different clients won’t be the same. The business relevance and statistical power of each analysis will differ with each client’s data.

Prediction market vendors may also need to familiarize themselves with the statistical learning methods necessary to fully utilize these rich datasets. So what’s the solution? First, move to a software-and-consulting model. By ‘consulting,’ I don’t mean ‘consulting on how to implement the market.’ I’m talking about helping the client solve its problem using a variety of data, including prediction market data.

Second, the vendors also need to pitch prediction markets as more than a forecasting tool. People in the business world commonly identify as data junkies — probably more so than they identify with the ‘wisdom of crowds’ ethos. It is unclear how much companies really care about accurate forecasting anyway.

-

Robin Hanson:

Info Value = the added accuracy the markets provide relative to other mechanisms, times the value that accuracy can give in improved decisions, minus the cost of maintaining the markets, relative to the cost of other mechanisms.

A highly accurate market has little value if other mechanisms can provide similar accuracy at a lower cost, or if few substantial decisions are influenced by accurate forecasts on its topic.

-

Bo Cowgill:

Although PM experiments make a company look ‘cool,’ disclosing specific high profile applications isn’t necessarily in the firm’s interests. I can vouch for this based on my own experience. Of a firm’s major stakeholders, we don’t know who would find a particular application unfair or irresponsible. This is particularly true if the application will create winners and losers — as would often be the case with high value markets.

Robin Hanson:

[...] I meant trying to field the highest value applications. That is naturally measured in accounting terms – value minus cost. Measures of popularity or familiarity would not at all be the same thing.

Bo Cowgill:

[...] I did not say anything about the popularity of the markets. I proposed measuring the number of employees working in a job that has a prediction market on it. This is a function of the firms decision to implement markets on a wide variety of relevant topics for the business.

I know this is a heuristic, and there would be greater value in trying to measure the value added in dollars. However, a monetization study is likely to be 1) more time consuming to produce, and 2) just as unreliable.

We aren’t likely to get honest answers from people by asking them to estimate the value of additional information at various specific hypothetical moments in the past. People are not good at making these type of estimations, especially when there is no incentive to get it right (and a lot of reasons to get it wrong).

On the costs side: Even if we had good data about how much individual employees were spending on the site browsing, etc — it is unclear how to price such their time. Is it work, or is it leisure? Should we model this as employees having an hourly rate (even though most are salaried)? I know you had some thoughts on this before which I don’t remember (do feel free to share), but I’m not convinced these issues can be resolved in a persuasive way.

Once people realize what’s going on with the methodology of such research, they’ll realize what a totally hackable and unreliable study it is — and it will lose its persuasive value at Google as well as externally.

In summary: Doing a value-of-information calculation on prediction markets itself does not seem to offer very much information value. Because of the low rigor, it would offer little additional persuasive value at a great cost.

Robin Hanson:

I can’t have much optimism about a business practice whose proponents aren’t even willing to try to offer a cost-benefit calculation. You could count how many employees had ever gone to a TQM meeting, but that wouldn’t tell you if TQM is valuable or not.

[Here's Robin Hanson's website. For your information (if you are a newbie), Robin Hanson is the most advanced researcher in the field of prediction markets. He co-invented the modern-day prediction markets, the concept of decision markets, and a new marked design, the Market Scoring Rule.]

-

Voila. Thanks for your attention. Feel free to publish a comment, just below… or blog about it —and I’ll update this post with some of your thoughts.

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

Decision markets could say whether this merger is good for shareholders, by estimating the combined stock price given a merger, and given no merger. Similarly, decision markets could say whether this merger is good for these firms’ customers, by estimating the price and/or quantity of web ads given a merger, and given no merger. This might help convince regulators to approve the merger.

My main doubt here is whether ad price and quantity are good enough measures of the merger’s social benefits – what other outcomes could such markets estimate, to speak more clearly? And this is a very clear demonstration that these companies are just not serious about finding the highest value applications of prediction markets.

Cross-posted from Overcoming Bias.