Midas Oracle .ORG » polls

The relative advantage of prediction markets (over conventional means of forecasting, namely polls and statistical models) is remarkably… SMALL.

Chris F. Masse — Fri, 22 Jan 2010 08:27:38 +0000

In a new study, Daniel Reeves, Duncan Watts, Dave Pennock and I compare the performance of prediction markets to conventional means of forecasting, namely polls and statistical models. Examining thousands of sporting and movie events, we find that the relative advantage of prediction markets is remarkably small. [...]

Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms. Nevertheless, it is certainly possible that there are forecasting applications where either the relative advantage of markets is larger, or that such differences in performance are consequential. Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

Prediction Without Markets — by Sharad Goel, Daniel M. Reeves, Duncan J. Watts, David M. Pennock

http://www.cam.cornell.edu/~sharad/papers/pred-wo-markets.pdf

Robin Hanson’s comment:

Whoever said that every prediction market would always be more accurate than any other mechanism? [*] I’d say they are more-accurate more often than they are less-accurate, compared to mechanisms with similar resources. And your plots at the bottom look like they are testing calibration, not accuracy.

Read Daniel Reeves’s answer to Robin Hanson.

[*] Google, doc.

Midas Oracle = About + Archives + Authors + Best + Contact + Information Technology + Links + Probabilistic Predictions + Post-Mortem Predictions + Prediction Exchanges + Prediction Software + Publish Your Post + How To Publish

One hundred poll respondents lied to InTrade.

Chris F. Masse — Fri, 11 Sep 2009 10:28:41 +0000

3. What would you likely use as an alternative if InTrade.com were no longer available?

I probably wouldn’t use an alternative. –> 65%

It is a lie. They would use BetFair USA instead —or another service. Come on.

And Chris Masse of Midas Oracle is not a “symbiotic detractor”, John. Blame yourself for the occasional slamming in the world’s #1 prediction market blog. I am a fair guy. All the slams published here are deserved. Take it graciously —as the other ones do.

Professor Christopher Wlezien on his research on how election prediction markets compare to polls.

Chris F. Masse — Wed, 02 Sep 2009 09:11:38 +0000

Professor Christopher Wlezien on his research on how election prediction markets compare to polls:

Markets vs. Polls as Predictors: An Historical Assessment of US Presidential Elections – (PDF file) – by Robert S. Erikson and Christopher Wlezien

The Prediction Market Police

Chris F. Masse — Fri, 28 Aug 2009 05:32:16 +0000

I should have listened to Panos Ipeirotis…!!!…

Simple Healthcare Observations

Jason Ruspini — Thu, 27 Aug 2009 01:49:58 +0000

As always, Nate Silver does an excellent job of processing the news, but his recent analysis of government health insurance polling neglects a crucial part of the question: willingness to pay. Why take pains to accurately portray a public policy if we don’t also frame it in terms of costs?

When asking about costs, it’s also enlightening to quantify them. Consider this awkward citation made by former Senator Tom Daschle in Critical:

A New York Times/CBS News poll conducted in February 2007 found that the majority of Americans wanted the federal government to guarantee health insurance to every citizen. Sixty percent of the respondents, including 62 percent of independents and 46 percent of Republicans, said they’d be willing to pay more in taxes to ensure that every American is covered.

Sounds compelling, and then you read the next sentence:

Half said they’d be willing to pay as much as $500 a year.

Huh? So, roughly speaking, the average person thinks it’s a good idea but does not want to pay more than a pittance, comparable to a couple additional months of FICA withholding, to make it happen? (And this for universal insurance.) Daschle of course realizes this tension but doesn’t readily address it. To me this is the mundane, selfish fundamental issue that underpins the vigor of the resistance to expanding coverage. Enough people are satisfied with their current coverage/situation and know that change will come with a cost. It is not principally an inadequacy in Obama’s “messaging” skills that can be usefully diagnosed over a laptop and a Starbucks Venti. It’s caveman politics.

Daschle does provide the useful warning that a bill’s complexity is directly proportional to the difficulty of passage. While moving towards single-payer could reduce prices, the presence of a public option in a bill creates a nightmarish legislative logjam.

Despite the emotional resonance of the coverage problem, the financing problem is more fundamental, otherwise coverage can’t be sustained. Begin to address the financing problem now, and try to expand coverage in another bill. Of course this is easier said than done as interests don’t want to risk being left out in the cold. If only there were another way for concentrated interests to meet each other half-way….

So much for Prof Panos and his data religion

Chris F. Masse — Fri, 05 Jun 2009 06:38:04 +0000

People answer bullshit when polled by Prof Panos and other stats collectors.

Flawed New Hampshire polls = Non-accurate New Hampshire prediction markets

Chris F. Masse — Tue, 28 Apr 2009 16:50:15 +0000

The most comprehensive analysis ever conducted of presidential primary polls:

“a handful of methodological missteps and miscalculations combined to undermine the accuracy of predictions about presidential primary winners in New Hampshire and three other states.”

Via Mister the Great Research Scientist David Pennock –who is an indispensable element of the field of prediction markets.

As I blogged many times, prediction markets react to polls… See the addendum below… – [UPDATE: See also Jed's comment.] – Prediction markets should not be hyped as crystal balls, but simply as an objective and continuous way to aggregate expectations. So, if you think of it, their social utility is much smaller than what the advocates of the “idea futures”, “wisdom of crowds” or “collective intelligence” concepts told us. Much, much, much, much smaller… They all make the mistake to put accuracy forward. (By the way, somewhat related to that issue, please go reading the dialog between Robin Hanson and Emile Servan-Schreiber.)

Addendum

California Institute of Technology economist Charles Plott:

What you’re doing is collecting bits and pieces of information and aggregating it so we can watch it and understand what people know. People picked this up and called it the “wisdom of crowds” and other things, but a lot of that is just hype.

New Hampshire – The Democrats

The Hillary Clinton event derivative was expired to 100.

New Hampshire – The Republicans

The John McCain event derivative was expired to 100.

Nate Silver of FiveThirtYeight.com on “The Interview Show” —recorded November 5th, 2008

Chris F. Masse — Fri, 17 Apr 2009 20:12:14 +0000

Nate Silver of fivethirtyeight.com on “The Interview Show,” recorded November 5th, 2008:

Nate Silver on November 3rd, 2008:

Nate Silver on August 21, 2008:

Damped polls are superior to prediction markets as election predictors.

Chris F. Masse — Sat, 07 Mar 2009 17:56:51 +0000

Are Political Markets Really Superior to Polls as Election Predictors? – (PDF file) – by Chris Wlezien and Robert Erikson – 2007

Abstract

Election markets have been praised for their ability to forecast election outcomes, and to forecast better than trial-heat polls. This paper challenges that optimistic assessment of election markets, based on an analysis of Iowa Electronic Market (IEM) data from presidential elections between 1988 and 2004. We argue that it is inappropriate to naively compare market forecasts of an election outcome with exact poll results on the day prices are recorded, that is, market prices reflect forecasts of what will happen on Election Day whereas trial-heat polls register preferences on the day of the poll. We then show that when poll leads are properly discounted, poll-based forecasts outperform vote-share market prices. Moreover, we show that win-projections based on the polls dominate prices from winner-take-all markets. Traders in these markets generally see more uncertainty ahead in the campaign than the polling numbers warrant—in effect, they overestimate the role of election campaigns. Reasons for the performance of the IEM election markets are considered in concluding sections.

Conclusion

This paper has tested the claim that the Iowa Electronic Market offers superior predictions of election outcomes than the snapshots from public opinion polls. By our tests, the IEM election markets are not better than trial-heat polls for predicting elections. In fact, by a reasonable as opposed to naïve reading of the polls, the polls dominate the markets as an election forecaster. This is true in the sense that a trader in the market can readily profit by “buying” candidates who, according to informed readings of the polls, are undervalued. Moreover, we find that market prices contain little information of value for forecasting beyond the information already available in the polls. Where then do the markets go wrong? To begin with, consider the vote-share market. The histories of market prices show that traders tend to hold persistent beliefs about the vote division that contradict the polls and that these persistent beliefs are often wrong. Incorrect beliefs get corrected only in the last days before the election, when the polls are difficult to ignore. The winner-take-all market tracks the vote-share market but compounds its errors by overvaluing long-shot candidates’ chances of victory, as if the market expects more campaign surprises than occur in reality. The existence of persistent mistakes in the vote-share market compounded by the degree of uncertainty about the vote-share estimates makes the winner-take-all market a particularly poor forecasting tool. Based on the experience of the IEM, if the polls show a candidate to hold a decisive lead but the market is unconvinced, bet on the polls. It should be noted that our daily poll projections are themselves rather crude instruments. Our robotic trading programs are informed by a flat prior, relying solely on the current polls and the days until the election but nothing more. Even when we compare market prices to the weekly average of poll-based forecasts, our instrument is primitive in that the week’s polls are not weighted for relative recency. But further perfection of our forecasting model from the polls would only advance our central argument. If we were to apply more rigorous modeling to obtain a properly weighted average of current polls and earlier polls, the victory of poll forecasts over the market forecast presumably would be more secure. One could argue that the results are drawn from a limited number of election years from a toy market with thin volume and limits on trader spending. With time, the IEM record could improve, and there is some suggestion that it has. Full-blown markets like Tradesports.com [or InTrade.com or BetFair.com] might in the end achieve an efficiency that so far has eluded the Iowa Electronic Market. Additionally, studies like the present one can suggest improved strategies to traders, which in turn improve the efficiency of election markets. Since our results are confined to a few runs of the toy Iowa market, some might claim a “so what” reaction. To such claimants, an important reminder is that the allegedly uncanny performance of the Iowa market has been touted as the primary evidence for the supposed superiority of election markets over the polls as an information source. The Iowa election market’s performance has not been so special after all. For now, our results suggest the need for much more caution and less naïve cheerleading about election markets on the part of prediction market advocates.

Damped polls outperform prediction markets.

Chris F. Masse — Sat, 07 Mar 2009 17:25:30 +0000

Forecasting Principles:

Damping polls

Evidence from the literature shows that polls, in particular early in the campaign, are not reliable in predicting election outcomes but tend to overestimate the extent to which a candidate leads. To deal with these uncertainties, we added a damping factor to the RCP poll average. Damping is used to make forecasts more conservative in situations involving high uncertainty.

Our poll damping is based on research conducted by Campbell (1996) who showed that polls have to be discounted in order to achieve more reliable forecasts. Performing a regression analysis on historical poll data for the elections from 1948 to 2004, he derived a formula for discounting the polls according to their distance from Election Day. Campbell provided Polly the formula, along with a list of damping factors that vary by the number of days left before the election.

Currently, polls are discounted with a damping factor of ${factor}. Applying this factor, we calculate Polly’s discounted poll based forecast thus:

Polly’s poll based forecast = ((Latest RCP polling average – 50) * (1 – [damping factor])) + 50 = ((46.1 – 50) * (1 – 0.17)) + 50 = 46.8

Latest RCP polling average 46.1
Damping factor 0.17
Polly’s poll based forecast 46.8

Thus, our poll damping discounts a candidate’s lead in the two-party vote, depending on the days left prior to election. The further away the election day, the larger the damping.

Such damped polls have been shown to outperform sophisticated forecasting approaches like prediction markets. Comparing damped polls to forecasts of the Iowa Electronic Markets, Erikson and Wlezien (2008) showed that the damped polls outperformed both the winner-take-all and the vote-share markets.

Thanks to Andreas Graefe for the link.

Are Political Markets Really Superior to Polls as Election Predictors? – PDF file

For now, our results suggest the need for much more caution and less naïve cheerleading about election markets on the part of prediction market advocates.

Previously: The truth about prediction markets