Flawed New Hampshire polls = Non-accurate New Hampshire prediction markets

No Gravatar

The most comprehensive analysis ever conducted of presidential primary polls:

&#8220-a handful of methodological missteps and miscalculations combined to undermine the accuracy of predictions about presidential primary winners in New Hampshire and three other states.&#8221-

Via Mister the Great Research Scientist David Pennock &#8211-who is an indispensable element of the field of prediction markets.

As I blogged many times, prediction markets react to polls&#8230- See the addendum below&#8230- – [UPDATE: See also Jed’s comment.] – Prediction markets should not be hyped as crystal balls, but simply as an objective and continuous way to aggregate expectations. So, if you think of it, their social utility is much smaller than what the advocates of the &#8220-idea futures&#8221-, &#8220-wisdom of crowds&#8221- or &#8220-collective intelligence&#8221- concepts told us. Much, much, much, much smaller&#8230- They all make the mistake to put accuracy forward. (By the way, somewhat related to that issue, please go reading the dialog between Robin Hanson and Emile Servan-Schreiber.)

Addendum

California Institute of Technology economist Charles Plott:

What you&#8217-re doing is collecting bits and pieces of information and aggregating it so we can watch it and understand what people know. People picked this up and called it the &#8220-wisdom of crowds&#8221- and other things, but a lot of that is just hype.

New Hampshire – The Democrats

The Hillary Clinton event derivative was expired to 100.

Dem NH Clinton

Dem NH Obama

Dem NH Edwards

New Hampshire – The Republicans

The John McCain event derivative was expired to 100.

Rep NH McCain

Rep NH Romney

Rep NH Huckabee

Rep NH Giuliani

No TweetBacks yet. (Be the first to Tweet this post)

Yet another guy, writing about prediction markets in the mainstream media, who does not master what he is talking about.

No Gravatar

Via Adam Siegel of Inkling Markets,

John McQuaid of Wired.

  1. It&#8217-s incoherent to start a rant against prediction markets by this upbeat line, &#8220-Prediction markets can be spookily accurate.&#8221-
  2. He blames the New Hampshire upset on poor liquidity. Where is the scientific evidence of that? Total invention by our good friend Barry Ritholtz. The New Hampshire prediction markets were wrong because the advanced, primary indicators (the polls) were wrong. As simple as that. [For why the polls were wrong, see: The New York Times, Zogby, Rasmussen, Gallup…]
  3. Prediction markets &#8220-have a lot of political junkies but few real insiders or outsiders, so they&#8217-re not very good at catching something the polls might miss.&#8221- Hummm&#8230- Most of the &#8220-real insiders&#8221- don&#8217-t keep scoops for themselves (if and when they have some), they are too happy to act as a source for some thirsty journalists or bloggers, so as to have their name printed somewhere. Hence, the political junkies would be able to aggregate any kind of extraordinary information &#8212-if that were to happen.
  4. How could the prediction markets &#8220-get out ahead of conventional wisdom&#8221-? It&#8217-s impossible, other than by reversing our psychological arrow of time (remembering the future, instead of the past). At the contrary, the job of the prediction markets is to quantify exactly that so-called &#8220-conventional wisdom&#8221-. They won&#8217-t go further, and we&#8217-re happy to run with that, because, that way, we are not prisoner of the bias of a handful of experts. Plus, prediction markets give us an objective probability of event outcome &#8212-a thing that individual experts can&#8217-t give us.

The excerpt below is good enough, though:

[…] But forecasting also needs more so-called noise traders, who do business with almost no information. Noise traders boost accuracy by increasing volume and the potential profits of informed traders. Diversity helps, too. If you can get different types of people to play, experts say, not only do you get a bigger pool and more information, but differing random guesses will cancel each other out, leaving real signals to rise above the noise. Plus, if you have a critical mass of investors with a variety of backgrounds, locations, and interests, they are less likely to move as a herd. […]

Previous blog posts by Chris F. Masse:

  • A second look at HedgeStreet’s comment to the CFTC about “event markets”
  • Since YooPick opened their door, Midas Oracle has been getting, daily, 2 or 3 dozens referrals from FaceBook.
  • US presidential hopeful John McCain hates the Midas Oracle bloggers.
  • If you have tried to contact Chris Masse thru the Midas Oracle Contact Form, I’m terribly sorry to inform you that your message was not delivered to the recipient.
  • THE CFTC’s SECRET AGENDA —UNVEILED.
  • “Over a ten-year period commencing on January 1, 2008, and ending on December 31, 2017, the S & P 500 will outperform a portfolio of funds of hedge funds, when performance is measured on a basis net of fees, costs and expenses.”
  • Meet professor Thomas W. Malone (on the right), from the MIT’s Center for Collective Intelligence.

Ubber finance blogger Barry Ritholtz believes in magic. He believes that, with more volumes on the event derivative markets, comes the Omniscience -capital O.

No Gravatar

Our good friend Barry Ritholtz.has persuaded himself that our real-money prediction markets suffer from an irremediable and fatal problem: liquidity on political event derivative markets is too thin for smart Wall Street people like him to take their market-generated probabilities seriously. Barry Ritholtz is keen to tout oranges&#8211-apples comparisons: the NYSE volume versus the Obama&#8211-Clinton volume at InTrade. It&#8217-s a bullshit argument, but he managed to persuade some gullible journalists writing for some clueless mainstream media that thin liquidity was responsible for the New Hampshire upset &#8212-and else.

Barry, if you had 1,000,000,000 trades on the New Hampshire prediction market, you&#8217-d still have an inaccurate prediction. The polls were wrong, and there&#8217-s nothing &#8230- NOTHING&#8230- that the InTrade and BetFair traders could have done to get this election right. Get over it, Barry. Traders are not magicians. :-D

[For why the polls were wrong, see: The New York Times, Zogby, Rasmussen, Gallup…]

Professor Koleman Strumpf explains the prediction markets to the countryland people.

No Gravatar

Ah, Kansas&#8230- Barbecues, pickup trucks, rednecks, country music, and&#8230- the local FOX News.

Despite that one blip on the radar [New Hampshire], Strumpf said futures are still the best way to predict the way things will go from here.

Spot the SIDEBAR (which is not located on the sidebar, actually), and click on the little square, just below &#8220-video&#8221-, to watch the report.

The BetFair blog claims a worldwide victory.

UPDATE: I just got it that Professor Leighton Vaughan-Williams&#8217-s story and the BetFair compound chart published on top of his story should be understood independently from each other, as this chart was pasted there by the BetFair blog editor.

&#8212-

Winston Churchill

Professor Leighton Vaughan-Williams on the official BetFair blog:

[&#8230-] Those taking the same advice on Tuesday evening [2008-01-15 = date of the Michigan primary] were similarly well rewarded as well-backed Mitt Romney stormed into clear favouritism in the markets and a comfortable victory at the polls. After a blip in the New Hampshire Democratic primary the old certainties – that election favourites tend to win elections – was re-established.

As in the Republican New Hampshire primary, the polls and pundits had declared the race between Senator McCain and Governor Romney as a toss-up while the betting markets pointed to a comfortable victory in both cases for the eventual winners. Once again, in the battle of the polls, pundits and markets, the power of the betting markets to assimilate the collective knowledge and wisdom of the crowd had prevailed. [&#8230-]

No BetFair charts are provided. Bad prediction market journalism.

UPDATE: The compound chart was under my very nose:

Michigan BeFair

&#8212-

UPDATE: I just got it that Professor Leighton Vaughan-Williams&#8217-s story and the BetFair compound chart published on top of his story should be understood independently from each other, as this chart was pasted there by the BetFair blog editor.

&#8212-

UPDATE: The BetFair blog has added a new label on the infamous compound chart&#8230-

Compound chart - BetFair blog fiasco

&#8212-

NEXT: Did the BetFair blog use trading data from InTrade to hint at BetFair&#8217-s accuracy??

&#8212-

For your information, I re-publish below the InTrade charts of the last 3 primary races (Wyoming excluded). [BetFair and NewsFutures do not provide on their site the charts of expired contracts. I could ask them later, though.]

&#8212-

Iowa

&#8212-

The Democrats.

The Barack Obama event derivative was expired to 100.

Dem Iowa Obama

Dem Iowa Clinton

Dem Iowa Edwards

&#8212-

The Republicans

The Mike Huckabee event derivative was expired to 100.

Rep Iowa Huckabee

Rep Iowa omney

Rep Iowa McCain

&#8212-

New Hampshire

&#8212-

The Democrats

&#8212-

The Hillary Clinton event derivative was expired to 100.

Dem NH Clinton

Dem NH Obama

Dem NH Edwards

&#8212-

The Republicans

&#8212-

The John McCain event derivative was expired to 100.

Rep NH McCain

Rep NH Romney

Rep NH Huckabee

Rep NH Giuliani

&#8212-

Michigan

&#8212-

The Democrats

The Hillary Clinton event derivative was expired to 100.

MI Dem Clinton

MI Dem Obama

MI Dem Edwards

MI Dem Field

&#8212-

The Republicans

The Mitt Romney event derivative was expired to 100.

MI Rep Romney

MI Rep McCain

MI Rep Giuliani

MI Rep Field

Source: InTrade


Author Profile&nbsp-Editor and Publisher of Midas Oracle .ORG .NET .COM &#8212- Chris Masse&#8217-s mugshot &#8212- Contact Chris Masse &#8212- Chris Masse&#8217-s LinkedIn profile &#8212- Chris Masse&#8217-s FaceBook profile &#8212- Chris Masse&#8217-s Google profile &#8212- Sophia-Antipolis, France, E.U. Read more from this author&#8230-


Read the previous blog posts by Chris. F. Masse:

  • Are David Pennock’s search engine prediction markets the worst marketing disaster since the New Coke?
  • Midas Oracle is incontestably [*] the best vertical portal to prediction markets.
  • Comment spam paid by Emile Servan-Schreiber of NewsFutures-Bet2Give
  • BetFair Games needs a Swedish provider to develop its gambling offerings.
  • When Markets Beat the Polls – Scientific American Magazine
  • Robin Hanson has some fanboy in India. Great. Tiny caveat: The parroting Indian writer does not acknowledge Robin Hanson by name.
  • Molecular Nanotechnology

New Hampshire is just another Pentagon moment for political prediction markets

No Gravatar

I think all the excitement over the New Hampshire prediction market prices is creating another &#8220-Pentagon moment&#8221- for the concept. A lot of attention is being paid to prediction markets, and it looks bad, but ultimately it will turn out just fine.

Consider that the criticism is a product of high expectations &#8212- people are complaining that, contrary to expectations, the markets were not better than the polls. People are noticing that prediction markets are not perfect. People all over the place are arguing about prediction markets.

It may look bad, but to me it appears as if there is a broad base of awareness and relatively high expectations out there that wasn&#8217-t evident in the press six months ago. Furthermore, a lot of people who hadn’t thought too much about prediction markets have now had the topic thrust in front of them.

It will turn out more than “just fine” for prediction markets – the market for prediction markets will continue to grow. It may look like a very public black eye, but that is because political prediction markets are now in the fight.

Prediction markets are forecasting tools of convenience that feed on advanced indicators.

No Gravatar

Why were the political prediction markets so wrong about Barak Obama and Hillary Clinton in New Hampshire?

&#8230-asks Slate&#8217-s Daniel Gross &#8212-via Mister Usability (Alex Kirtland), who needs to go and get his own gravatar.

So, I&#8217-ve been watching the action in one of the political futures markets this evening, Intrade. And the action in this prediction market has reinforced my opinion that these are less futures markets than immediate-past markets. The price movement tends to respond to conventional wisdom and polling data- it doesn&#8217-t lead them.

Throughout the day and into the early evening, while polls were still open, Democratic investors, mimicking the post-Iowa c.w. and polls, believed Obama was highly likely to be the Democratic nominee. The Obama contract was trading in the lows 70s, meaning investors believed he had a 70 percent chance of being the nominee, while Hillary Clinton contracts were in the 20s. […] At 6 p.m., this market had written Hillary Clinton&#8217-s entire presidential campaign off. At 9:30 p.m., it was calling a dead heat. What caused investors to change their minds so drastically in the space of a couple of hours? A few data points that went against the day&#8217-s prevailing conventional wisdom and polls. […]

See also Niall O&#8217-Connor&#8217-s assessment:

I am looking forward to the post New Hampshire Caucus, when all you prediction market advocates crawl out from under your stones. For the record at one point the market on Intrade and Betfair was suggesting that Obama had a 95% probability of winning the caucas- whilst Intrade had him at 77% to win the nomination.A case perhaps of both the foolery of crowds and, the market biting back.

New Hampshire will go down as the Black Wednesday of prediction markets and unless there is now objective transparent debate (as opposed to the usual biased sabre rattling) – prediction markets will be dead in the water.

My answer to Dan Gross&#8217- legitimate question and to Niall O&#8217-Connor&#8217-s snarky comment:

  1. Prediction markets are forecasting tools of convenience that feed on advanced indicators. When those advanced indicators are wrong, the prediction markets are wrong.
  2. If you prefer the polls or the pundits, your call &#8212-but polls and pundits were also wrong, this time, right? Required reading for mister Niall O&#8217-Connor: &#8220-New Hampshire&#8217-s Polling Fiasco&#8221- + &#8220-Analysis: pundits eat crow&#8220-.
  3. The ultimate forecasting tool would be a way to reverse our psychological arrow of time &#8212-so as to remember the future instead of the past. Only science-fiction writers and some imbecile ( :-D ) believe in that.
  4. The prediction market approach is to stick with the markets, on the long term. Take their successes. Take their failures. Unlike Donald Luskin and Markos Moulitsas, Chris Masse will not turn against the prediction markets when they fail punctually. What counts is the long series.
  5. My first point should be included in the prediction markets approach definition, in my view, but others (like economist Michael Giberson) might have different opinions.
  6. With respect to my first point, I bet that the prediction markets will never replace the polls as the forecasting tool of choice for political analysts &#8212-on that particular point (but not on a myriad of others), I break away from Justin Wolfers&#8217- irrational exuberance and I side with Emile Servan-Schreiber of NewsFutures (my preferred play-money prediction exchange). Prediction market reporting will have a function, indeed (as suggests Justin Wolfers), but not the dominant function.
  7. Going forward, prediction market journalism should emphasize relative accuracy (as opposed to absolute accuracy) &#8212-that is, comparing prediction markets with polls and pundits, which is what Robin Hanson has said from day one. Our good friend Niall O&#8217-Connor has difficulty to compute that, apparently. He should eat more fish. :-D

&#8212-

Justin Wolfers:

In a few years, we may regard the second half of the 20th century as the aberration in which the press used polls rather than markets to track political races,” Justin Wolfers, a business professor at the University of Pennsylvania’s Wharton School, wrote in an e-mail message. “And in the 21st century, we may return to the habits of the early 20th century, reporting on political races through the lens of prediction markets rather than polls.

Emile Servan-Scheiber:

1) The traders themselves are the first to look at the polls to inform their trades. So the polls are here to stay.

2) Our recent experience in Western Europe seems to indicate that the superior accuracy of markets over polls when predicting elections may be a U.S. artifact that isn’t so easily reproducible elsewhere. I’ve discussed this with Forrest Nelson of IEM [Iowa Electronic Markets], and apparently, ever since the Truman-Dewey polling debacle of 1948, U.S. pollsters have adopted a policy of reporting mostly raw numbers rather than projections based on sophisticated secret formulas, so they can’t be accused of manipulating opinion. However, raw numbers are notoriously unreliable when based on small samples, and Western European pollsters never report them, preferring instead to publish projections based on historically-informed statistical formulas. What we’ve observed in France and Holland is that it it’s very hard to beat the accuracy of such projections.

[I don’t make mine Emile Servan-Schreiber’s second point, but that’s a minor.]

&#8212-

InTrade&#8217-s expired prediction markets:

&#8212-

New Hampshire

&#8212-

The Democrats

&#8212-

The Hillary Clinton event derivative was expired to 100.

Dem NH Clinton

Dem NH Obama

Dem NH Edwards

&#8212-

The Republicans

&#8212-

The John McCain event derivative was expired to 100.

Rep NH McCain

Rep NH Romney

Rep NH Huckabee

Rep NH Giuliani

&#8212-

Iowa

&#8212-

The Democrats.

The Barack Obama event derivative was expired to 100.

Dem Iowa Obama

Dem Iowa Clinton

Dem Iowa Edwards

&#8212-

The Republicans

The Mike Huckabee event derivative was expired to 100.

Rep Iowa Huckabee

Rep Iowa omney

Rep Iowa McCain

&#8212-

Source: InTrade

&#8212-

[A more complete prediction market reporting should have included expired contracts from NewsFutures and BetFair. Sorry for that. Note that InTrade-TradeSports is the only exchange to offer a “closed contacts” section.]

&#8212-

NEXT: Prediction Markets 101 + Who did best in explaining the prediction markets to the lynching crowd? + After the New Hampshire fiasco, 16 people came to defend the prediction markets, so far. + The prediction markets deserve a fair trial. + Prediction Markets = the greatest time-saving invention of this century

&#8212-

Leading political indicators

No Gravatar

American politics does not suffer from a shortage of polls. Zogby. Gallup. Rasmussen. SurveyUSA. Mason-Dixon. Polimetrix&#8230- In an information-glutted world, what matters is not the supply of sources, but the ability to glean trustworthy information from the larger swath of poor data.

Different polling organizations have different strengths and weaknesses. Some use &#8220-tight screens&#8221- to scope out likely voters- others simply sample registered voters, without making any attempt to tighten the survey base to &#8220-likely voters.&#8221- Tight screening is especially crucial to gauge the true state of a primary, when committed base opinion can diverge significantly from less engaged moderate voters, and more importantly, influence those moderates over time to converge to the more partisan perspective. Some use human interviewers, although recently that has given way to IVR (Interactive Voice Recording) polls (the kind where a computer talks to you and asks you to &#8220-press 1 if you will definitely support X, 2 if probably&#8230-&#8221-)

I have found tight-screen, IVR polling to be the most reliable. IVR not only has no marginal cost, but it eliminates all the biases resulting from trying to give the most pleasant-sounding answer possible (the &#8220-sexy grad student effect&#8221- that exaggerated Kerry&#8217-s margin by 15 points in Pennsylvania 2004 exit polling, for example). IVR possible responses can also be randomly rotated from respondent to respondent to eliminate recency biases (first and last responses in a list exaggerated because those are at the forefront of a person&#8217-s memory of the list, not because s/he will vote that way).

The poster-child of IVR tight-screen polling success is Scott Rasmussen&#8217-s Rasmussen Reports. I have only tracked them over the last two election cycles (2004 and 2006), but considering that 2004 was a GOP wave and 2006 a Democratic wave election, I think the data is sufficient to form a valid judgment. Rasmussen&#8217-s track record is simply stupendous. It predicted 49 out of 50 states in 2004 correctly, usually within two percentage points of the actual outcome. In 2006, Rasmussen achieved similarly impressive results &#8212- all the more impressive when you consider that most polling models tend to err in favor of one party or the other. (&#8220-Likely voter&#8221- models tend to favor Republicans, and registered voter-based models tend to exaggerate Democratic strength.)

My other favorite sources include Gallup and Mason-Dixon. Gallup comes closer to the &#8220-registered voter&#8221- model than the tighter Rasmussen model, so Gallup usually lags tighter-screen polls. By election eve, however, the two models usually converge. Gallup&#8217-s election-eve congressional generic vote is hands-down the best in the business. However, their numbers for party primaries have poor predictive value, because they don&#8217-t make much effort to hunt down likely voters.

Differing survey methods can yield very different results. Rasmussen has long shown a much closer Democratic nomination race than most established, &#8220-registered voter&#8221- pollsters &#8212- most recently, it showed a 32-32 tie between Clinton and Obama, with Edwards wallowing 15 points behind. Gallup&#8217-s last numbers tightened drastically to a 31-26 race between Clinton and Obama (Gallup&#8217-s numbers are also hard to compare with Rasmussen&#8217-s because Gallup includes Gore).

Many smart Democrats, notably MyDD&#8217-s Chris Bowers, believe that Gallup and others are mistakenly including lots of &#8220-low information voters&#8221- who simply lag the opinions and thought processes of more-attuned Democratic partisans.

Now that more establishmentarian polling firms are coming in line with Rasmussen&#8217-s results, one can infer that the likely voter/ Chris Bowers theory has gotten the better of the argument.

A survey of pollsters wouldn&#8217-t be complete without knowing which ones to stay away from. Stay away from Zogby and CNN polling. James Carville&#8217-s and Stan Greenberg&#8217-s DemocracyCorps polling outfit is not trustworthy, either &#8212- for example, when they doubled the percentage of blacks in an October 2006 survey sample to bump the Democrats&#8217- generic advantage by 5 points, to reinforce the Democratic narrative of a building wave.

Lastly, partisan pollsters in a competitive election season should always be taken with a grain of salt &#8212- they will use heuristic subtleties to create the best impression possible for their party&#8217-s candidates. Strategic Vision, a Republican outfit, deserves a three- or four-point handicap. Franklin Pierce generated a dubious Romney result for New Hampshire right after its lead pollster, Rich Killion, went to work for the Romney campaign. Such polls should be trusted only as a last resort.

For those of us who wish to divine movements in politics futures, discerning trustworthy data from bad data is paramount. Poll-rigging is the high art of Washington, DC, and as any interest group &#8212- or candidate &#8212- knows, it&#8217-s easier than easy to produce a poll that diverges wildly from reality, if the heuristics are threatening enough.

(cross-posted from my blog, The Tradesports Political Maven)