PredictionBook is a new prediction tracking site, as described at LessWrong, which:

lets you make predictions and then track your calibration – see whether things you assigned a 70% probability happen 7 times out of 10.

The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.

The site is a bit confusing–I have no idea what it means to “judge” others’ predictions of unresolved claims, but the idea of prediction tracking is an old one, touted with caveats for example by Robin Hanson (see here, here and here) and others, and it’s good to see a concrete attempt.

I’ve only looked at a few predictions, but one with many participants presents an interesting point of comparison with a roughly equivalent prediction market — “Barack Obama will serve a second term as president of the United States continuous with his present term.” I assume this prediction would count as true if Obama served one second of a 2nd term, rather than serving out a full 2nd term, but such are unmoderated claims.

The interesting part is that prior to my prediction, 24 were made ranging from 55% to 95%. My prediction?

mlinksva
estimated 53% and said “.647*.820 last trades for 2012.DEM.NOM.OBAMA and PRESIDENT.DEM.2012 on Intade.”
1 minute ago

Yep, that should be .820*.647 (the two claims respectively, though this doesn’t change the estimate) and I misspelled Intrade–ah well.

MBlume, the creator of the claim, estimated 80% and commented “I intentionally made this prediction prior to checking the intrade market.” One other user, Matt, said “I’ll track InTrade on this one.”, estimating 63%. However, Matt must have only checked PRESIDENT.DEM.2012 — four days ago when the estimate was made the last Intrade price was 63.1. 2012.DEM.NOM.OBAMA was 82.0. If he had wanted to track Intrade, he would have estimated .820*.631 = 52%.

For at least this one prediction on this one site, estimates are pretty crazy without the corrective of trading against crazy estimates.

I wonder if anyone has examined the accuracy of probabilities implied by multiple prediction market contracts such as the above? I would be fairly startled if such were inaccurate given the performance of the underlying prices that implied probabilities are derived from through multiplication, though perhaps biases seen with very low- or high-probability events would be exaggerated.