If there were actual contracts for these predicates or at least an interface for their implied prices, that might have an outside shot of helping with some of these questions.

But I have no clue how to even attempt to model this theoretically, as this requires to be able to somehow quantify information completeness.”

I agree that we will need to be able to estimate the accuracy of a prediction market before it is run ‘live’. However, I do not think we will be able to analyse “similar” types of markets to do so. More likely, we will have to determine the calibration for virtually identical prediction markets. For example, an EPM to forecast quarterly sales of a corporate division. The same EPM would be run every quarter and the participants would be very similar in each one.

Finally, I agree with the information completeness issue. Presently, other than using calibration accuracy as evidence of information completeness, I’m not sure how we might quantify this very subjective attribute. Few, if any, seem to be working on this issue.

On a first sight, yes, it seems more like a capricious academic game.

However, being able to tell “how well-calibrated a PM on event X will be, given the characteristics of the event X” is an important question to answer. It explains whether it makes sense to deploy a market.

Qualitatively, I have a feeling on how to approach this (open committee vs closed committee, secrecy etc). It is even possible to empirically estimate the “degree of calibration”, for every event-type, using past markets. Then we can run, say, regressions to see how the degree of calibration is affected by various characteristics of the event.

But I have no clue how to even attempt to model this theoretically, as this requires to be able to somehow quantify information completeness.

I don’t know whether it will be possible to quantify information completeness. I suspect that we will not be able to do so. However, I do think, logically, that when information completeness is substantial, the market outcomes will be fairly highly correlated. I still don’t think this will mean anything, where the market is comprised of discrete options (as discussed previously).

In enterprise prediction markets, it should be possible to estimate the level of calibration for similar markets, over time. I doubt we will be able to generalize the findings to other types of prediction markets, but we should be able to develop calibration indices for each type of market.

To align our vocabularies: With accuracy, in this setting, I think that Jed (and myself) mean the degree of calibration. So, for “open” committees where information flows, the degree of calibration will be high. For highly-secretive committees, the degree of calibration will be low.

For a perfectly calibrated market, the scatterplot of price-vs-”fraction-of-times-outcome-occurs” is a perfect diagonal. For a market that generates random decisions, the scatterplot of price-vs-”fraction-of-times-outcome-occurs” is an almost set of points in the rectangle.

The statement about “secretive (or not) committees” refers to how accurately we can predict their outcome, using prediction markets. Not how well a committee can predict its own decision.

In other words, the question is: Given the characteristics of the event, can we predict the degree of calibration of the corresponding prediction market?

You correctly mention that it depends on the information completeness. But can we operationalize and quantify the notion of information completeness? Can we quantify how information completeness affects the degree of calibration of the corresponding prediction markets?

Then, he mentions that secretive (or not) committees will not be as accurate as traditional open prediction markets. Perhaps I’m misreading this, but it makes no sense to compare these two methods.

Panos asked whether there might be a more principled method of determining the accuracy of prediction markets. I’ve written a bit about this. All that needs to be done is to fill in the details.

I’ve posted more detailed comments on my blog:

http://torontopm.wordpress.com/2009/10/20/more-public-prediction-market-failures/

Tricky, but not impossible. HSX has an excellent record with both Oscar winners as well as the nominations (which are more difficult because there is no limit on possibilities).

“Another example of things difficult to divine: Selection of a VP in US presidential election.”

Very true. In my mind that would be generally equivalent to a Nobel Peace Prize winner. (Very small committee choosing from a very wide pool of candidates in near-total secrecy.)

But again, there is a spectrum of accuracy in running markets on these decisions.

Last year, everybody predicted Mickey Rourke, and he didn’t get it, coz Hollywood saw it as a jerk. Hollywood does not see things as Main Street does. (See the Polanski affair, for instance.)

So trying to divine Hollywood’s future decisions is tricky.

Another example of things difficult to divine: Selection of a VP in US presidential election.

I definitely agree with your Academy Awards as another good example. Unfortunately, I’m not aware of any work along these lines. I got started thinking about all this to try and distinguish why some markets would be worse than others, and came up with some of the ideas above.

It would be great to see some of this modeled, but I’m not sure how to go about it with existing data, as there is so little of it in many cases (such as one IOC decision every two years). A good first step would be to develop a comprehensive list of factors that could affect accuracy.

Potentially you can add Academy Awards (Oscars) as a even more open committee. I am wondering what is the proper modeling approach to capture this phenomenon in a more principled manner. Are you aware of any work along these lines?