Final InTrade v. Zogby Showdown Results

No Gravatar

Superdelegate chicanery notwithstanding, Obama has won his party&#8217-s nomination, so my head-to-head contest between a major pollster (Zogby) and major prediction market (Intrade) will be coming to a close.

Unsurprising to those who know a little bit about the scholarship, economics, and/or track record of prediction markets, the traders of Intrade provided us much better data this election season than the respondents to the Zogby polls.

Standings
Wins Losses Ties Pct Contender Avg Eve Prob
7 3 11 59.5% Intrade 71.3%
3 7 11 40.5% Zogby 40.7%

Schedule
Score Date State Party Intrade Zogby Winner Intrade Pct Zogby Pct
7-3-11 6-May IN Dem Clinton 2-way-tie Clinton 85% 42%
6-3-11 6-May NC Dem Obama Obama Obama 90% 50%
6-3-10 22-Apr PA Dem Clinton Clinton Clinton 82% 47%
6-3-9 4-Mar OH Dem Clinton 2-way-tie Clinton 70% 45%
5-3-9 4-Mar TX Dem Obama 2-way-tie Clinton 57% 44%
5-2-9 5-Feb NJ Rep McCain McCain McCain 96% 52%
5-2-8 5-Feb NJ Dem Clinton 2-way-tie Clinton 67% 43%
4-2-8 5-Feb NY Rep McCain McCain McCain 98% 53%
4-2-7 5-Feb GA Dem Obama Obama Obama 96% 48%
4-2-6 5-Feb MO Dem Obama Obama Obama 63% 47%
4-2-5 5-Feb CA Rep McCain Romney McCain 56% 40%
3-2-5 5-Feb CA Dem Obama Obama Clinton 52% 46%
3-2-4 29-Jan FL Rep McCain 2-way-tie McCain 51% 33%
2-2-4 26-Jan SC Dem Obama Obama Obama 90% 38%
2-2-3 19-Jan SC Rep McCain McCain McCain 56% 29%
2-2-2 19-Jan NV Dem Obama Clinton Clinton 54% 42%
2-1-2 15-Jan MI Rep McCain 2-way tie Romney 54% 27%
2-0-2 8-Jan NH Dem Obama Obama Clinton 91% 39%
2-0-1 8-Jan NH Rep McCain McCain McCain 82% 34%
2-0-0 3-Jan IA Dem Obama 3-way tie Obama 54% 28%
1-0-0 3-Jan IA Rep Huckabee 2-way tie Huckabee 53% 28%

Lord willing, I will have another showdown season, but will most likely choose another pollster. Whereas Intrade listed contracts and probabilities for every state primary, Zogby only provided election eve updates for only 21 of the 87 held to date. Rasmussen stands out as a worthy pollster, but I would be happy with any candidate referrals from my readers as well.

Cross-posted from Caveat Bettor.

Previous blog posts by Caveat Bettor:

  • Land-Ocean year-to-date temperatures 0.35 Celsius over baseline
  • Intrade lists Global Warming Contracts!
  • Intrade beats Zogby on Super Tuesday
  • Super Tuesday Showdown: Intrade v. Zogby
  • The Democrat SC Showdown: Intrade v. Zogby
  • Zogby beats Intrade in predicting Nevada caucus winner Clinton.
  • The GOP SC and Dem NV Showdown: Intrade v. Zogby

18 thoughts on “Final InTrade v. Zogby Showdown Results

  1. Chris F. Masse said:

    Hi Cav,

    Congrats for you work. For your information, I do subscribe to the Memeorandum feed and I saw Gallup popped up there many times in this primary election season. I didn’t see much the other pollsters, as far as I can remember.

    PS: Memeorandum, like TechMeMe, computes the bloggers’ links to worthy web pieces —in that, it’s comparable to Google’s PageRank system.

  2. Chris F. Masse said:

    The machine created two posts (#6854 and #6855). This one is the #6855. I deleted the #6854.

    If you known what happened, tell me, so I can understand whether it’s a bug, or a flaw in the WordPress user interface, misleading the blogger —which is something I have remarked and blogged about here yesterday.

    http://www.midasoracle.org/200…..wordpress/

  3. Jason Carver said:

    Long time reader, first time poster. Wow, very interesting data, it’s great to see these comparisons. I would love to see the analysis framed a different way, though. The current analysis seems to embrace the idea that a market is “wrong” when it implies a 55% chance of event occurrence and it doesn’t occur. Implicitly condoning this as a relevant interpretation (via win/loss count) serves to promote people who use that same idea against prediction markets, as in articles like this:

    Slate: The Fizz-dom of Crowds

    The market binning strategy tells a fairly different tale…

    markets in the 50-60 % range:

    9

    agree with proposition:

    5

    actual average probability:

    56%

    markets in the 60-90 % range:

    6

    agree with proposition:

    6

    actual average probability:

    100%

    markets in the 90-100 % range:

    6

    agree with proposition:

    5

    83%

    Here, you can see the market was mostly “wrong” when it should have been. Conversely, one could argue that the market was actually incorrect in how many times the outcome was successful, even with lower probability estimates in the 60-90 range (although there were too few markets in this data to be sure). There were also too few data points to know if the NH reversal was truly a 1/10 probability or whether the market was overconfident.

    If the world is to understand prediction markets, it seems that the most informed people (all you fine folks running and contributing to midas oracle) should form a united front in explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen. The world needs to understand that this fallacious argument is equivalent to saying any probability calculation can be invalidated by a single event (for example, 67% is still the correct probability for rolling >2 on a die, even after seeing a single roll of a 1).

    Edit to add paragraph breaks

  4. David Pennock said:

    Great effort and analysis, thanks.

    Next time you might try Real Clear Politics or Wikipedia.

    Actually, it may not be too late to analyze these for the nomination contest too.

  5. Chris F. Masse said:

    @Jason Carver: “explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen”… We have been doing that on the explainer you’ll find on the frontpage, and in each political PM story like the last one.

    To separate a paragraph, we put a “hyphen”, as you see above. I’ll see look into that issue when I update the “theme” (layout) of this blog.

  6. Chris F. Masse said:

    @David Pennock: Speaking of RCP, Cav, do google The Numbers Guys at the WSJ for an analysis of pollsters and averaged polls.

  7. Jason Carver said:

    @Chris Masse: thanks for the tip regarding paragraph breaks.

    Explainers are useful, but I still think propagating the W/L concept at all is confusing and undoes a lot of the good done by the home page explanations.

  8. Caveat Bettor said:

    Thanks for the great comments Chris, David and Jason.

    Jason–you have identified a mismatch in my contest, namely, that the Intrade contracts are “all-or-none” payout derivative, rather than a “spot settlement” payout which would reflect the actual percentage of votes received.  If it were the latter, then the mismatch would be eliminated, as the Zogby and Intrade predictions could be matched against the actual percentages.

    But because this scenario was not possible, in part because of the complexity of and risks to settlement,

    I tried to design an experiment that would measure success of each predicted trial.

    Delegates for each state are assigned a pro-rata basis, but everyone is pretty much concerned with which candidate got the most delegates for each state.  My experiment attempts to measure for this.  The Intrade candidate contract with the highest value for each primary is the predicted winner, while for Zogby, it is the candidate with the highest percentage above the 3% confidence interval stated by Zogby.

    What this meant was that a candidate did not need to project over 50% to be the predicted winner of a state contest; merely, the candidate had to project ahead of his or her competition.

    Hope this helps, Cav

  9. Jason Carver said:

    Cav, thanks for the added info.  Interesting stuff!

  10. Chris F. Masse said:

    Hi Cav,

    Congrats for you work. For your information, I do subscribe to the Memeorandum feed and I saw Gallup popped up there many times in this primary election season. I didn’t see much the other pollsters, as far as I can remember.

    PS: Memeorandum, like TechMeMe, computes the bloggers’ links to worthy web pieces —in that, it’s comparable to Google’s PageRank system.

  11. Chris F. Masse said:

    The machine created two posts (#6854 and #6855). This one is the #6855. I deleted the #6854.

    If you known what happened, tell me, so I can understand whether it’s a bug, or a flaw in the WordPress user interface, misleading the blogger —which is something I have remarked and blogged about here yesterday.

    http://www.midasoracle.org/200…..wordpress/

  12. Jason Carver said:

    Long time reader, first time poster. Wow, very interesting data, it’s great to see these comparisons. I would love to see the analysis framed a different way, though. The current analysis seems to embrace the idea that a market is “wrong” when it implies a 55% chance of event occurrence and it doesn’t occur. Implicitly condoning this as a relevant interpretation (via win/loss count) serves to promote people who use that same idea against prediction markets, as in articles like this:

    Slate: The Fizz-dom of Crowds

    The market binning strategy tells a fairly different tale…

    markets in the 50-60 % range:

    9

    agree with proposition:

    5

    actual average probability:

    56%

    markets in the 60-90 % range:

    6

    agree with proposition:

    6

    actual average probability:

    100%

    markets in the 90-100 % range:

    6

    agree with proposition:

    5

    83%

    Here, you can see the market was mostly “wrong” when it should have been. Conversely, one could argue that the market was actually incorrect in how many times the outcome was successful, even with lower probability estimates in the 60-90 range (although there were too few markets in this data to be sure). There were also too few data points to know if the NH reversal was truly a 1/10 probability or whether the market was overconfident.

    If the world is to understand prediction markets, it seems that the most informed people (all you fine folks running and contributing to midas oracle) should form a united front in explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen. The world needs to understand that this fallacious argument is equivalent to saying any probability calculation can be invalidated by a single event (for example, 67% is still the correct probability for rolling >2 on a die, even after seeing a single roll of a 1).

    Edit to add paragraph breaks

  13. David Pennock said:

    Great effort and analysis, thanks.

    Next time you might try Real Clear Politics or Wikipedia.

    Actually, it may not be too late to analyze these for the nomination contest too.

  14. Chris F. Masse said:

    @Jason Carver: “explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen”… We have been doing that on the explainer you’ll find on the frontpage, and in each political PM story like the last one.

    To separate a paragraph, we put a “hyphen”, as you see above. I’ll see look into that issue when I update the “theme” (layout) of this blog.

  15. Chris F. Masse said:

    @David Pennock: Speaking of RCP, Cav, do google The Numbers Guys at the WSJ for an analysis of pollsters and averaged polls.

  16. Jason Carver said:

    @Chris Masse: thanks for the tip regarding paragraph breaks.

    Explainers are useful, but I still think propagating the W/L concept at all is confusing and undoes a lot of the good done by the home page explanations.

  17. Caveat Bettor said:

    Thanks for the great comments Chris, David and Jason.

    Jason–you have identified a mismatch in my contest, namely, that the Intrade contracts are “all-or-none” payout derivative, rather than a “spot settlement” payout which would reflect the actual percentage of votes received.  If it were the latter, then the mismatch would be eliminated, as the Zogby and Intrade predictions could be matched against the actual percentages.

    But because this scenario was not possible, in part because of the complexity of and risks to settlement,

    I tried to design an experiment that would measure success of each predicted trial.

    Delegates for each state are assigned a pro-rata basis, but everyone is pretty much concerned with which candidate got the most delegates for each state.  My experiment attempts to measure for this.  The Intrade candidate contract with the highest value for each primary is the predicted winner, while for Zogby, it is the candidate with the highest percentage above the 3% confidence interval stated by Zogby.

    What this meant was that a candidate did not need to project over 50% to be the predicted winner of a state contest; merely, the candidate had to project ahead of his or her competition.

    Hope this helps, Cav

  18. Jason Carver said:

    Cav, thanks for the added info.  Interesting stuff!

Leave a Reply

Your email address will not be published. Required fields are marked *