[...] So what is the “right†way to evaluate probabilistic predictions? There is no single absolute best way, though several tests are appropriate, and probably can be considered stronger tests than the calibration test. In our paper “Does Money Matter?†we use four evaluation metrics:
1. Absolute error: The average over many events of lose_PR, the probability assigned to the losing outcome(s)
2. Mean squared error: The square root of the average of (lose_PR)2
3. Quadratic score: The average of 100 – 400*(lose_PR)2
4. Logarithmic score: The average of log(win_PR), where win_PR is the probability assigned to the winning outcomeNote that the absolute value of these metrics is not very meaningful. The metrics are useful only when comparing one predictor against another (e.g., a market against an expert).
My personal favorite (advocated in papers and presentations) is the logarithmic score. [...]
Meta
-
Recent Posts
- Interview with Adam Lashinsky — [VIDEO]
- Why some people are more innovative — [VIDEO]
- Forbes editor deciphers Steve Jobs’s Apple. — [VIDEO]
- Jason Ruspini rebuts Eric Zitzewitz on the regulation of political prediction markets. — [COMMENT]
- Eric Zitzewitz petitions the CFTC in favor of real-money prediction markets about politics. — [TEXT]
- Global warming is a big scam. — [LINK]
- A Swarm of Nano Quadrotors — [VIDEO]
- The Tragedy of the Commons — [VIDEO]
- Guy Kawasaki on Steve Jobs — [VIDEO]
- Inside Apple — [VIDEO]
- Mitt Romney’s taxes — [LINKS]
- A critique of Apple’s multimedia iBooks. — [LINK]
- Does Apple lack “generosity”? — [LINKS]
- Apple Education Push — [LINKS]
- Water Crystals — [DOCUMENT]
- Apple’s e-book software will allow publishers to make textbooks more interactive. — [LINKS + VIDEO]
- Alain Soral is France’s most dangerous intellectual… (dangerous for the French plutocrats, that is). — [VIDEO]
- Computers thru time — [CHART]
- NASA has finally understood the theorical basis of LENR (low-energy nuclear reactions). — [VIDEO]
- Why Samsung is no Apple — [VIDEO]