-

-


-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

Blip.TV -

-

-

Via Daniel Horowitz (Business and Technology Consultant)

Blip.TV — (FLV file)

-

-

-

It’s cool.

-

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.

-

DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

-

PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]

We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.

The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]

Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]

The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]

Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.

Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]

Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]

In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]

-

FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.

-

APPENDIX:

-

Bo Cowgill’s precision point on micro-geography:

Below you can see a snapshot of trading in one of our offices. The areas where employees are making profitable decisions is green, and the areas where employees are making unprofitable decisions is red. There are about 16 profitable traders in that big green blotch in the middle!

-

More information from our previous blog post on the Google paper

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

BetFair CEO David Yu:

We’re delighted to welcome Bo Cowgill in HammerSmith. Bo had a fantastic career at Google, working, in the end, under Google’s Chief Economist Hal Varian. Throughout his years at Google, Bo acquired precious knowledge about the inner working of search engine technology, which will be helpful to BetFair-TradeFair for fighting annoying externalities: Internet rumors and bombastic bloggers.

Bo Cowgill (Internet Meister at BetFair):

Being a number one myself, I’m delighted to work now for the world’s number one prediction exchange, after my work at the world’s number one search engine. I can only sympathize for BetFair. I have been myself the butt of many Internet jokes (associating me with a “Grandizer” I’ve never met); numerous inaccuracies have been published over the years on blogs about my person and my work at Google; my EXCELLENT Google paper has been totally ignored by the prediction market bloggers (PDF file); and my intellectual fight with prediction market ubber-expert Robin Hanson has been totally misrepresentated (the truth is that I won the discussion in the end). At BetFair, I’ll work hard to crash bloggers and forums’ rankings in Google’s web search results. Count on me, big time.

Nick Garner (SEO expert at BetFair):

I’m happy a world-renowned Google expert is joining our SEO team in HammerSmith. I have so far optimized the BetFair sites for Alta Vista (that’s the first search engine listed in my A-to-Z list of search engines), and I am looking forward to reach the “G” letter very soon with Bo’s help.

Mark Davies (Spin Doctor In Chief at BetFair):

Enhancing our SEO team is an integral part of our strategy to take over the Internet from the bombastic bloggers and the unmoderated forums, and this move sits well alongside other recent BetFair initiatives (the BetFair blog, the sports and politics zones, the dynamic compound charts, etc.). Very soon, the Internet will resemble a giant BetFair megaphone, and that shall make the world a happier place. [Sorry to be so short, for once, but I have to leave you, real quick. I must arrive at the horse track VIP gallery before Princess Margaret.]

-

Previously: Google is THE problem, but BetFair should discover the solution.

-

-

VIDEO — Bo Cowgill on Google’s enterprise prediction markets — O’Reilly Money:Tech

Blip.TV — (FLV file)

-

-

-

-

I’ll first publish some excerpts from the paper on Google’s enterprise prediction markets (set up by Bo Cowgill, Doug Banks, Patri Friedman, Ilya Kirnos, Piaw Na and Hal Varian), and then I’ll list all the interesting discussion points that this paper has generated on the Web…

-

Here’s Bo Cowgill’s face pic, for those who don’t know him yet:

Bo Cowgill – LinkedIn profile

-

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.

-

DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

-

PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]

We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.

The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]

Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]

The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]

Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.

Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]

Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]

In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]

Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]

-

FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.

-

APPENDIX:

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

Via Zubin Jelveh, Bo Cowgill’s precision point on micro-geography:

Below you can see a snapshot of trading in one of our offices. The areas where employees are making profitable decisions is green, and the areas where employees are making unprofitable decisions is red. There are about 16 profitable traders in that big green blotch in the middle!

-

Eric Zitzewitz in The New York Times:

Peoples’ primary job isn’t to trade these commodities. What we are picking up is communication on ‘low priority topics.’ But that’s how creative ideas come about.

-

Bo Cowgill in Network World:

If you let people bet on things anonymously, they will tell you what they really believe because they have money at stake. This is a conversation that’s happening without politics. Nobody knows who each other is, and nobody has any incentive to kiss up. [...]

As a philosophical matter, Google likes to pack people in tight… so they can share information. As a company gets larger, people don’t always adhere to the founding tenets, or if they’re in a big building they’ll spread out because it’s more comfortable. This was something where we could say ‘there’s value and we can measure that, and we can compare it to what it’s like when people e-mail.’

-

Bo Cowgill (answering Mat Fogarty’s remark):

[...] I don’t think that short-aversion explains the optimistic bias revealed in the Google markets. Because of the IEM-like structure of the markets, no shorting was necessary to make a pessimistic bet. If you wanted to bet pessimistically, you could simply buy the pessimistic outcome. In other words: The optimistic bias was not only saw an aversion to shorting optimistic outcomes. It was also an aversion to buying pessimistic ones. We did observe a general aversion to shorting, which is bad for the market’s efficiency and accuracy. Better terminology could overcome this. More on these two biases in a forthcoming post.

-

The 12 principles that guide programming at Google:

1. All developers work out of a ~single source depot; shared infrastructure!
2. A developer can fix bugs anywhere in the source tree.
3. Building a product takes 3 commands (“get, config, make”)
4. Uniform coding style guidelines across company
5. Code reviews mandatory for all checkins
6. Pervasive unit testing, written by developers
7. Unit tests run continuously, email sent on failure
8. Powerful tools, shared company-wide
9. Rapid project cycles; developers change projects often; 20% time
10. Peer-driven review process; flat management structure
11. Transparency into projects, code, process, ideas, etc.
12. Dozens of offices around world => hire best people regardless of location

The Google paper made a social graph of these code reviews and compared trading habits among circle of co-reviewers. See Table #2 and Table #3.

-

Economist Michael Giberson has more on “Prediction markets and the flow of information inside organizations“.

[...] With private, anonymous trading, the choice to disclose information via the market dramatically revises the mental decision calculus involved. The trader is rewarded if right and penalized if wrong, but in either case the disclosure and net reward is a private matter rather than social event. (At least until you brag about it around the water cooler.) [C]andor should be rewarded and incentives designed to encourage it. Prediction markets provide incentives for candor. Not only that, but over time the traders with useful candor are encouraged by accumulated gains, while blowhards find their accounts diminished. It is true that prediction market prices present relatively limited signals. Prices may go up or down, but they never say why. But with a signal, at least someone knows to start asking “why” and that is better than not knowing.

-

Concurring with Justin Wolfers, here’s Tim Harford on the flow of information around Google’s physical office spaces:

[...] The results were striking. Clear correlations existed between the trading behaviour of certain groups of employees. But they were not explained by shared interests or by social connections. Having the same immediate boss only explains a little about information flows. No, it is the office layout that matters: people who sit near each other tend to know the same things, as evidenced by making similar trades on the prediction markets. Social and professional proximity matters very little for the flow of information: physical proximity is almost everything. Specialists in organisational behaviour have known for a while that people tend to interact much more with those who sit nearby, but it has never been clear whether that was just social grooming. Now we know that real information is flowing. We keep being told that because of cheap, ubiquitous communication technology, distance is dead. But if there was ever a company that we should expect to exemplify that idea, surely it was Google. This research suggests that it is as important as ever to be sitting in the right place.

-

Reading Tim Harford, an objection popped up in my mind. If we had ben following this micro-geography model all along, then we would never have had open-source software (e.g., Linux, Mozilla FireFox, WordPress) and open knowledge (e.g., Wikipedia, group blog BoingBoing). In all these instances, collaboration has been done over the Internet by people who don’t know each other in person —although they happen to meet, once in a while . And yet these open projects have contributed greatly, in my view, to the making of our global digital civilization.

-

Enterprise prediction market consultant Jed Christiansen:

I’ve seen this in other markets I’ve run, as well. At the beginning traders are all over the place, with a fair bit of volatility and inaccuracy. But pretty quickly this settles out. I believe it’s a combination of “dumb” traders getting frustrated and stopping, and all traders becoming more sensitive to the risks they’re taking. [...]

Google used a CDA model for their markets. This created arbitrage opportunities, when the sum of the bid prices was more than $1, and when the sum of the ask prices was less than $1. The authors found 1,747 instances of the former, and 495 instances of the latter. As they noted, this demonstrates an aversion to short selling contracts. What was interesting to me was how the market reacted to this. The median duration of any of these arbitrage opportunities was just two minutes, demonstrating that they were correctable. Even better is this: “One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle.” All it takes is one person and a little programming knowledge to make the market much more efficient and liquid! I would never discourage this kind of behaviour. [...]

Bo Cowgill’s reacting about the organizational-behavior argument:

[...] This is not the first reaction along these lines. I am perplexed by the response. I can understand why other companies may not want to replicate our analysis of information flows. Perhaps it wouldn’t be worth the effort. Perhaps they would get identical results. And perhaps the company wouldn’t have the all the necessary data.

However, I expected that people could easily see value in the analysis of granular trade-by-trade data — especially if that data is joined with data about traders and outside events happening at the moment of the trades. We described one very generic application of this approach, but you can imagine much more actionable and company-specific ones.

I will mention one: The data contains real-time metrics on the distribution of knowledge and attitudes within a firm at a highly granular level. You can get metrics on for specific of the firm, for specific classes of employees and for specific topics. You can do this for either customers or employees, and have the metrics for any moment in time. The quality of these metrics will be extremely strong, because participants have been incentivized to reveal their true expectations.

Our analysis spoke in very general terms about the flow of information between Google employees — we don’t reference specific groups or draw distinctions between them — which is where a lot of actionable data was. Trade-by-trade data can reveal characteristics of specific working groups: What they know, how they feel, how they process and share information and how all of that changes over time. I didn’t try to put any of this in the paper because the conclusions would be sensitive, and I thought this application was pretty obvious to anybody who understood our methodology.

UPDATE: Our findings about the clustering of attitudes should also inform anyone who thinks that diversity is important for crowd-wisdom applications — as James Surowiecki famously suggests in The Wisdom of Crowds.

Our analysis suggests that groupthink primarily happens within language networks and small physical spaces (with social/professional networks playing a secondary role, and demographic networks playing a non-existent one). Remember that as you’re selecting your traders. If they already work/sit/chat together, the groupthink may already exist and the market won’t cure it.

Reacting on Jed Christiansen’s other post, Bo Cowgill adds:

I’ve also heard that other companies would find it impossible to analyze the interaction between their market and the organization. Why? Lack of data. Our analysis benefited from a wealth of internal data (including GPS coordinates of offices) that other companies don’t store.

You may be surprised at how much data average companies really have. For example, Google had social network surveys; many companies do not. However, many standard corporate applications (such as email, calendars, telephones and code reviews) contain implicit social networks that can be used in place of data gathered from surveys.

Or, consider this: I recently met with people from Google’s real estate management group. Turns out, they have records of the floorplans of Google’s offices in electronic format. Not only can someone use these records to find the distance between offices (without GPS coordinates) — you can also find the total area and perimeter of each office, which desks are open (cube-style) vs. enclosed, the walking distances between offices and more.

Surprised and impressed, I asked if it was typical for companies to have all of this information. The response was: “Any Fortune 1000 company would have this data about their offices.” Everyone in the room said his previous employer had the same data — typically managed through computer-aided facility management systems such as Archibus or Infor.

Bo Cowgill:

Some more remarks about applications that combine prediction markets and organizational data (org charts, social networks, seating locations). The obstacle to these applications is not a lack of data. Jed mentions privacy concerns — and if he thinks this is a big obstacle then I’d be interested in discussing his thoughts.

A bigger problem is that that current prediction market vendors and consultants cannot support these applications. At heart, these vendors are software engineers and salespeople at heart, not statisticians or data miners. They want to write one system that can support lots of clients. At conferences, one hears PM vendors complain about having to do “customization” work for clients.

This approach would not work for the applications I describe for two reasons:

1. The inputs for different clients won’t be the same. Each client’s organizational data will likely take a different structure. This makes it difficult for prediction market vendors to architect a single system that can served many clients (yet another challenge with integrating markets with other corporate IT services).
2. The outputs for different clients won’t be the same. The business relevance and statistical power of each analysis will differ with each client’s data.

Prediction market vendors may also need to familiarize themselves with the statistical learning methods necessary to fully utilize these rich datasets. So what’s the solution? First, move to a software-and-consulting model. By ‘consulting,’ I don’t mean ‘consulting on how to implement the market.’ I’m talking about helping the client solve its problem using a variety of data, including prediction market data.

Second, the vendors also need to pitch prediction markets as more than a forecasting tool. People in the business world commonly identify as data junkies — probably more so than they identify with the ‘wisdom of crowds’ ethos. It is unclear how much companies really care about accurate forecasting anyway.

-

Robin Hanson:

Info Value = the added accuracy the markets provide relative to other mechanisms, times the value that accuracy can give in improved decisions, minus the cost of maintaining the markets, relative to the cost of other mechanisms.

A highly accurate market has little value if other mechanisms can provide similar accuracy at a lower cost, or if few substantial decisions are influenced by accurate forecasts on its topic.

-

Bo Cowgill:

Although PM experiments make a company look ‘cool,’ disclosing specific high profile applications isn’t necessarily in the firm’s interests. I can vouch for this based on my own experience. Of a firm’s major stakeholders, we don’t know who would find a particular application unfair or irresponsible. This is particularly true if the application will create winners and losers — as would often be the case with high value markets.

Robin Hanson:

[...] I meant trying to field the highest value applications. That is naturally measured in accounting terms – value minus cost. Measures of popularity or familiarity would not at all be the same thing.

Bo Cowgill:

[...] I did not say anything about the popularity of the markets. I proposed measuring the number of employees working in a job that has a prediction market on it. This is a function of the firms decision to implement markets on a wide variety of relevant topics for the business.

I know this is a heuristic, and there would be greater value in trying to measure the value added in dollars. However, a monetization study is likely to be 1) more time consuming to produce, and 2) just as unreliable.

We aren’t likely to get honest answers from people by asking them to estimate the value of additional information at various specific hypothetical moments in the past. People are not good at making these type of estimations, especially when there is no incentive to get it right (and a lot of reasons to get it wrong).

On the costs side: Even if we had good data about how much individual employees were spending on the site browsing, etc — it is unclear how to price such their time. Is it work, or is it leisure? Should we model this as employees having an hourly rate (even though most are salaried)? I know you had some thoughts on this before which I don’t remember (do feel free to share), but I’m not convinced these issues can be resolved in a persuasive way.

Once people realize what’s going on with the methodology of such research, they’ll realize what a totally hackable and unreliable study it is — and it will lose its persuasive value at Google as well as externally.

In summary: Doing a value-of-information calculation on prediction markets itself does not seem to offer very much information value. Because of the low rigor, it would offer little additional persuasive value at a great cost.

Robin Hanson:

I can’t have much optimism about a business practice whose proponents aren’t even willing to try to offer a cost-benefit calculation. You could count how many employees had ever gone to a TQM meeting, but that wouldn’t tell you if TQM is valuable or not.

[Here's Robin Hanson's website. For your information (if you are a newbie), Robin Hanson is the most advanced researcher in the field of prediction markets. He co-invented the modern-day prediction markets, the concept of decision markets, and a new marked design, the Market Scoring Rule.]

-

Voila. Thanks for your attention. Feel free to publish a comment, just below… or blog about it —and I’ll update this post with some of your thoughts.

-

Using Prediction Markets to Track Information Flows: Evidence from Google – (PDF file – PDF file) – by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz – 2008-01-06

-

-

Freakonomics interview.

Hal Varian’s previous post.

Prediction markets and elections, run by Eric Zitzewitz of Dartmouth. Eric ran though a slew of data looking at sports and financial prediction markets and compared it to election prediction markets. Bottom line, sports and financial prediction markets are highly efficient and closely track actual outcomes, while election prediction markets tend to systematically underprice favorites. This is the way it is in the US. In the UK, where election prediction markets have existed for over a century, they are much more efficient. Two explanations for the US election markets’ relative inefficiency are low repetition vs. high repetition domains (sports and financials have large numbers of contests while elections have far fewer contexts), and that bettors in the US have far less collective experience betting on elections than their UK counterparts. Interesting stuff.

Greg Linden’s post (from which I lifted the sentence posted in the title above)

Inkling Markets’ Adam Siegel

Collective Intelligence Foo Camp

List of attendees.

Prediction markets came up as topic of interest #1 —at least online.

Wiki participation was low, though.

I was invited, but couldn’t make it, due to time pressure. From the e-mail they [*] sent me at the time:

CI Foo is, by design, far from a typical conference. The participants are the presenters–so be prepared to demo or speak. We’ll schedule some sessions ahead of time, and set up a process for participants to announce sessions they want to offer. Expect the kind of exchange that happens at the best conferences during breaks and late into the night, plus time for fun at Google’s legendary campus [the "GooglePlex"]. The result will be an informal but intense and eye-opening couple of days.

[*]

1. Gary Flake, Microsoft Live Labs;
2. Tom Malone, Director, MIT Center for Collective Intelligence;
3. Tim O’Reilly, O’Reilly Media;
4. Hal Varian, Google Chief Economist.

Freakonomics interview. (I did ask, but they didn’t listen.)

Hal Varian’s post.

The answer is yes. Hal has a small team now which I am on. Hal is my boss and cube mate.

Previously: Hal Varian becomes Google’s chief economist. + Meet Hal Varian, Google’s Chief Economist.

Via Greg Mankiw (whom you will have to read between the lines), this New York Times piece:

The View column will now be written by a rotating panel of outside economists. Besides Mr. Mankiw, it will include Alan Blinder, Judith Chevalier, Robert Shiller and Lester Thurow, as well as three of the economists who have been writing the Economic Scene column on Thursdays: Austan Goolsbee, Tyler Cowen and Robert Frank. (The fourth member of the Scene rotation, Hal R. Varian, is leaving to concentrate on his new role as Google’s chief economist.)

The Midas Oracle readers will remember that professor Hal Varian is the economics authority (revered by the Google executives) who consulted with Bo Cowgill’s 20% team on designing and pitching an internal prediction markets pilot. Professor Hal Varian commands respect and has made important contributions to Google’s core business.

—

Hal R. Varian interviewed by the Wall Street Journal on his new position at Google.

WSJ: What does the job entail?

Varian: During my time at Google we have built up a world-class group of quantitative analysts, and the economics team will complement these existing resources. Google has a great infrastructure for data analysis, and a management team that is very receptive to quantitative methods and willing to invest in this area. So what more could you ask for? In addition to working on analytics, I’ve also worked on various business strategy and public policy issues, and will continue to do so as the occasion arises. This set of issues will only get more important to Google as time goes on, so I expect that this will also involve a fair amount of my time.

Note: Bo Cowgill‘s official business title at Google is “Technical Data Analyst“.

—

Previous: Meet Hal Varian, Google’s Chief Economist.