Using prediction markets to support IT project management

The quotes that Ribstein draws from the Wall Street Journal article on Boeing highlight two problems – one with project management, and the other with senior management’s lack of information about the problem. From the WSJ:

Boeing Chairman and Chief Executive Jim McNerney said the company has sent manufacturing and procurement experts numbering “in the hundreds” to suppliers’ factories after discovering problems with the first 787 delivered to Boeing’s final assembly line. Those problems, including a serious lack of documentation on the work remaining to complete the first airplanes, drove the company’s decision this month to delay the first 787 deliveries for six months and to replace the head of the plane’s development efforts. Noting that Boeing was “surprised on the physical reality” of the condition of the first plane, Mr. McNerney said officials “realized we really need to work with [suppliers] to make sure we have better visibility” on the manufacturing process. “We need that data transparency across all of the build in order to execute the plan that we’ve laid out.“

Then Ribstein asks:

Could prediction markets reduce surprises and provide better corporate governance? Henry Manne thinks so. See Insider Trading: Hayek, Virtual Markets, and the Dog that Did Not Bark. An internal prediction market, say inside Boeing, might cull information from all nooks and crannies of the organization that might not otherwise be forthcoming.

Ribstein also suggests that had unregulated insider trading been allowed in Boeing stock, it may have also tipped off upper management to the problems. In Manne’s article, Manne makes similar claims for insider trading – not too surprising given his history on the issue – and then Manne suggests that prediction markets might do a better job than insider trading in this regard.

Obviously, in hindsight, Boeing had exactly the kind of problem that prediction markets can help resolve. Clearly all the information available to some people in the organization was not flowing smoothly to all the people who needed to know. Equally obviously, the cost of setting up such a market would have been much smaller that the costs the company is now facing. Given the high-level, resource-intensive intervention into 787 project management, apparently the surprising delays were judged to be a multi-million dollar problem. (Clearly, this guy was onto something in asking for airplane prediction markets. Bet2give arrived too late on the scene to help Boeing this time, but there is still time for more things to go badly, so maybe some value there if trading builds up.)

And the informational problem isn’t just at Boeing – Ribstein also mentions “Merrill’s $8.4 subprime hit.” A near endless list other examples could be plucked from the business headlines. There may be other organizational reforms or information technologies that would have also brought these problems to the attention of upper management in a more timely fashion. Are they better, or worse, than prediction markets?

Given the high stakes, any company that aspires to be a world-class operation should be asking these questions.

[I’ll post an overview of Manne’s article in a day or two. -MG]

[...] But [Consensus Point's David] Perry still recommends that prediction markets be used to answer high-value questions such as sales forecasts, ideal product features and future demand levels. [...]

Chris Masse recommends that prediction markets be used to answer questions where the other forecasting tools may have specific accuracy problems —e.g., the schedule aspect of project management.

• QUESTION: Who the hell are you? You came out of nowhere, in June 2007, and in the space of three short months, you’ve become a Midas Oracle fixture. What’s driving you to writing on prediction markets?

I’m an economist with a PhD out of George Mason University. My research at GMU revolved around electric power markets, but while working with Vernon Smith and other experimentalists at ICES, and I was exposed to prediction markets. I write about prediction markets because I think about them, and I think about prediction markets because I find them fun, interesting, and potentially useful.

• QUESTION: I got it that professor Robin Hanson was your next-door colleague, during your stint at George Mason University. How can you describe him, on a personal level? Is he aloof or is a good chap? [chap = British lingo]

After finishing my PhD I had a one-year research position at GMU, and because of space constraints at ICES I was put into one side of Robin Hanson’s office. (This was not his main office on campus, just the one he had on the Arlington campus.) In my experience Hanson is an ordinary and nice person, who also happens to be extremely bright, holds a number of beliefs that many people would find strange, is fairly passionate about his beliefs, and is unusually willing to consider alternatives to those beliefs.

• QUESTION: Please explain Robin Hanson’s invention, the Market Scoring Rules, in 10 sentences maximum.

A “scoring rule” is a formula used to assess the quality of forecasts. Because a scoring rule can be used to assess quality of forecasts, it can also be used to reward forecasters. Hanson realized (i.e., invented the idea) that a group of forecasters could sequentially share a common forecast, with a scoring rule used to reward forecasters for incremental improvements made to the forecast. Hanson further developed his idea by describing how a prediction market could employ a market scoring rule to serve as an automated market maker.

While different market scoring rules are possible, Hanson noted that the logarithmic market scoring rule has a number of desirable properties, and that approach appears to have been most commonly adopted.

• QUESTION: Are you paid by Adam Siegel (the Inkling CEO) to blog about their prediction exchange (betting exchange)?

Unfortunately no, I’m an amateur. (Maybe someday someone will pay me for this stuff. Make me an offer.) I blog about Inkling because I use Inkling and I try to blog about what I know.

(I also blog about economics, energy regulation, music, technology and many other topics at Knowledge Problem.)

• QUESTION: Why do you prefer speculating on Inkling Markets as opposed to NewsFutures or the play-money InTrade-TradeSports?

I think a couple of things attract me to Inkling – I like that it uses a LMSR-based automated market maker, I like the user interface (or at least, I did up until Friday morning; not too sure about the recent changes). To some extent, that fact that I’ve been somewhat successful keeps me coming back – if I fell off the “all time” top ten list at Inkling, maybe I would devote more energy elsewhere. I guess social rewards work. But I like the “do it yourself” aspect a lot, it gives the place something of a bazaar atmosphere.

Probably, I should venture out more and expand my horizons.

• QUESTION: I am very reluctant to cite any of the Inkling prediction markets on Midas Oracle, as long as I’m uncertain of the reputation of who expire those event derivatives. Agree, disagree?

I agree. Expiration and poor contract definition and market abandonment are all problems that trouble the markets. (Some of these problems are not exclusively the province of the do-it-yourselfers.) I’m beginning to agree with you that Inkling should adopt some sort of reputation or reward system for market managers. Maybe a ‘top ten’ managers list ranked by activity. At the least, Inkling could append a list of current and expired markets at the bottom of the market manager’s user profile page. I’ve noticed a few users that manually list their markets on their profile page, and that is a start.

• QUESTION: I believe that there are two big conspiracies in America. Number one, they want to silence Ron Paul. Number two, they’ll do everything they can to avoid citing BetFair, the world’s biggest prediction exchange (betting exchange). I’m asking you, if BetFair gets the second most liquidity on US political prediction markets, why doesn’t BetFair get cited just after InTrade-TradeSports (the market leader in North-America)? BetFair is never mentioned anywhere. Not fair.

BetFair gives every impression of being a success at sports betting, but sports betting generally doesn’t generate a lot of U.S. press unless corruption of some sort is involved. (And, by the way, BetFair was mentioned in U.S. media in conjunction with the ATP match-fixing suspicions.)

Does BetFair have a Ron Paul contract? I can’t find the political markets on www.betfair.com. Ah, click on “BetFair Sports” scroll down through the list of “All Markets”, and there lies “Politics” between “Rugby League” and “Poker.” Select “USA,” select “Republican candidates.” Aha! There he is at “24” in decimal odds (or, alternately, at 23/1 standard odds). A quick calculation indicates an implied 4.1 percent chance of receiving the GOP nomination.

I suspect American reporters (1) find it easier to locate the American political markets on Intrade, and (2) find prices directly interpretable as probabilities easier to report.

Also, the charts are hard to find (hidden as a link attached to the name, but the name is just black text against a white background and gives no indication that a chart and other information is available by clicking the name). If a reporter finds the market, clicks on the hidden link, and clicks the “Inverse Axis” box under the chart, then finally the reporter gets an easy-to-interpret implied probability chart.

While BetFair obviously works for experienced bettors, inexperienced visitors are given little help in discovering information that readily feeds into news stories. If BetFair wants the free publicity, it will need to be easier for reporters to get what they want.

• QUESTION: The PopSci PPX betting exchange has plenty of technology and science prediction markets. I love them. Do you foresee the generalist prediction exchanges taking on science and technology, one day? If yes, why, and, if not, why not.

Science and technology markets are harder to do than sports or election picks. Expiration can be difficult, because science is naturally messy. Lots of gray areas will have to be sorted out in the contract well in advance. In addition, interesting science and technology questions often will play out over years (or decades). It is not obvious that such markets would offer a money making opportunity for a prediction exchange, nor sustain active trader participation.

But PPX is experimenting in a play money environment in which it is cheap to explore and discover what works and what doesn’t. I suspect that the generalist exchanges will continue to try out new ideas too. Maybe these fine people will sort it out. I don’t think it will become a big business for anyone anytime soon.

• QUESTION: The Sim Exchange has recently stated that it will open product quality prediction markets, using MetaCritic as expiry source. What kinds of quality prediction markets would be socially valuable? Any ideas?

I like SimExchange, they seem to be working pretty hard (and pretty smart) to make their business work. The quality markets struck me as a great idea. If I had half a clue about video games, I’d trade there.

Ideas for socially valuable quality markets? Are you assuming that improving video game industry forecasts is not socially valuable? I would suggest, in addition to improving video game industry forecasts, maybe Chinese toy manufacturing quality markets would be useful. Maybe U.S. toy executives would like to set up product recall prediction markets.

Actually, product recall markets may be useful, though since they involve very low probability events, some thought would have to be put into the design to make them interesting and potentially rewarding to traders.

• QUESTION: What’s your definition of a prediction market (betting market)? Of a prediction exchange (betting exchange)?

I don’t have a good answer here. What makes a good definition depends on the purpose you have at hand.

• QUESTION: What’s the best academic paper you’ve ever read on prediction markets? What’s the most didactic paper on prediction markets you’ve ever read?

I like both Hanson’s “Combinatorial Information Market Design” and the Rhode and Strumpf work on historical markets and manipulation. But like a good definition, a good academic paper depends on the purpose you have at hand. “Best” to me depends on what issues I’m thinking about at the moment.

On the other hand, maybe someone like George Tziralis who has made an effort to do a literature review could provide an answer. I’ve read too narrowly.

• QUESTION: Do you really think that the economists have made the case that (non-sports) prediction markets are socially valuable? Don’t you think that more comparisons are needed, especially in specific instances where the other tools could be dysfunctional? [WARNING: Biased question that takes the view that we need more special prediction markets, e.g., on project management or on some areas of demand forecasting, that can show a macro difference with the predictions from the other tools.]

I certainly am of the view that more comparisons are needed.

And by the way, did I mention that I work for an economic consulting firm? You can send us money and we will do analysis. So far we specialize in energy markets, but, you know, make me an offer.

• QUESTION: If you were at the helm of Midas Oracle (thank God and all the Saints of Christianity, you’re not), what would you do to improve the coverage on prediction markets?

I think readers would benefit from a somewhat more postings offering substantive engagement with issues and somewhat fewer postings that lack prediction market substance. (When I say “readers would benefit”, what I mean is “I want.”) Not every post needs to be a Chris Hibbert explainer, but judging from the comments on Hibbert’s most recent posting there is an audience ready for high level substance.

I’d also like to see more practical reports. We need to entice users of internal corporate prediction markets (and other approaches to aggregating information) to tell us their stories. For how many more years will we have to link to Bo Cowgill’s September 2005 blog post?

What works, what doesn’t, what is hard to get right, what problems are people having, and so on. Of course, people in the business of selling solutions generally don’t want to admit to having problems. Still, if we are going to push the frontier of knowledge forward, we’ve got to know where the frontier is.

• QUESTION: Do you have a visionary thing to say about prediction markets? Some kind of formal pronouncement we will re-read here in two or three years to see whether it has stood the test of time (and, hence, whether you’re a real thinker who matters, like Robin Hanson).

I think it is likely that within three years the signers of the Hahn-Tetlock letter will get what they’ve asked for. There is still political and bureaucratic work to be done to get it accomplished, but I think that effort will succeed. While personally I’d favor a much bolder proposal, it will be hard enough for American politicians to sign on to something like Hahn-Tetlock.

By the way, this is another reason ordinary companies doing ordinary business things like forecasting sales should report publicly on their experiences. Prediction markets need demystification.

We’ve always felt this how [corporate prediction markets] may evolve: Instead of [..] a standalone marketplace, we expect to be fully integrated instead with an ERP system like SAP or Oracle, CRM systems like salesforce.com, or other mission critical systems of the same ilk. [I]f you think about it, the business processes prediction markets have seen success (financial forecasting, project management, product performance, competitive analysis, etc.) are many of the same inputs used in these systems. [Edits mine]

I completely agree. A corporate prediction market that talks or integrates with other corporate services confers tons of benefits. Among them:

• The market can talk with business intelligence software, corporate datawarehouses or document indexes to show relevant graphs, tables or files to traders. Better informed traders will improve predictions.
• The market can talk to HR databases to authenticate users and ensure that no employee signs up twice. In a large company, the HR databases can also help the market group employees into teams to facilitate competition among or within smaller populations.
  • Don’t forget: One day, the dream of employee bonuses for good trading might come true. On that day, the market will need to talk to the HR and compensation systems.
• Integrated markets can display where managers and employees are already looking. No need for an intra-firm marketing push to get people visiting the trading site. Placing the markets there also confers authority and legitimacy to the practice.

I could go on about this potential. The interesting question is: What does this mean for companies selling prediction market software?

1. Does this require the markets to be hosted completely within the firm? My impression is that the current generation of corporate prediction markets are hosted on outside servers. Access to the service is limited to those within an IP range or with a functioning login. For policy or technical reasons, these markets may not be able to talk to firmware behind the firewall.
2. If Inkling (or other firms) offered APIs, would clients use them? Conversely, would clients let Inkling use the APIs of their internal systems? (Ed: Do clients even have APIs for their firmware that can be used to talk with Inkling? Many don’t.)
3. How much connectivity with corporate services is necessary? A one-time dump of HR data into Inkling, or a one-time paste of JavaScript code into a dashboard page would be an easy way to integrate markets. How many of these opportunities exist? How feasible are they?
4. Firmware providers such as Oracle, SalesForce or MicroStrategy could be best positioned to deliver integrated PMs. Will they ever make an investment in developing or acquiring PM software? Would they be receptive to a sales pitch to acquire a startup?
5. Can corporate PMs (and corporate PM companies) flourish without a major firmware provider for distribution?

Going the firmware integration route doesn’t look easy for the current generation of PM startups. Do share your strategies for this (if you don’t consider it too sensitive).

Originally posted at my site.

He writes about corporate prediction markets being used for forecasting in matters of research and development (R&D), project management and product sale. Just a short thought, if I may. Prediction markets are not really the best tool for forseeing the years ahead of us —one quarter or one semester away is what they are good at.

Previous: Developing a Business Case for Prediction Markets + ROI of Prediction Markets for Businesses + Jed Christiansen, prediction market consultant