You may wish to propose a rule that, as you put it, “a [binary] prediction market would be accurate if it has been over 50% in the last time period”, and that rule might be easy to understand, and expedient, and easy to apply and draw headline-grabbing black and white conclusions from. But that rule would still violate logic and severely misconstrue the meaning of a probabilistic prediction.

I fully agree with the goal of trying to establish ways to assess the predictiveness of a single binary market’s price, but I don’t think your “for dummies” rule cuts it. Let’s dig deeper and ask around… let’s leverage the mind power of the MO audience !

However, on Midas Oracle, as you have seen, in addition to some postings about the meta-analysis done by Justin Wolfers and Eric Zitzewitz and the like, we also publish dynamic prediction markets (like whether Ms. Clinton will become the next US president). Once those prediction markets have expired, I think it is a public service to our readers to inform them about their expiry. And at this point, it’s all natural to say, oh that prediction market was wrong, or that prediction market was right.

It’s all natural to try to do such accountancy of the expired prediction markets, on an individual basis.

Now if your point is that on Midas Oracle, we should publish more meta-analysis, we should remind more often our readers about what really means a probabilistic prediction, or that we should remind more frequently our readers that what counts is not the individual prediction markets but the long series, yes. But I should have the right to publish information about the expiry of single prediction markets, and make a comment about each of them, and whether they were accurate or not, individually.

For the binaries, a prediction market would be accurate if it has been over 50% in the last time period. From there, there’s a graduation to introduce. The best way is to cite the percentage, so readers see whether the absolute accuracy was weak or strong.

The NewsFutures prediction market on Karl Rove signaled a 80% chance that he would stay at the White House. That was clearly not accurate.

You believe that the Karl Rove market was wrong. What value would it have needed to trade it two days ago in order to be “right”?

40%? 50%? 80%? 95%?