The methodology of prediction market event study — Multiple Causes Edition — REDUX

Keith Jacks Gamble on Michael Abramowicz’s Gamble on Gambling on Keith Jacks Gamble’s Super Bowl Analysis Highlights:

I agree that the simultaneity problem and the anticipation problem are confounding factors in assessing a player’s contribution to his team’s chance of winning. I’d like to say more on these issues.

(1) The Simultaneity Problem. Certainly, football is a team sport, and thus I suspect that cleanly identifying a player’s sole contribution play-by-play to his team’s chance of winning may be impossible. Certainly the performance of the offensive line has a ton to do with a team’s success. But even with standard statistics for measuring a player’s performance these issues remain. Quarterbacks get credit for completions, passing yards, and throwing touchdowns even though the coach’s play calling, the line’s blocking, the receiver’s route running, ect have everything to do with creating those numbers. Attributing actions to the primary actors on a play is a natural way to compute statistics when faced with the simultaneity problem. At least my net probability points statistic provides some weight to the game situation when measuring a player’s impact on a given play. Certainly, a 3 yard run on 3rd and 2 at the opponent’s 4 yard line means more than a 3 yard run on 3rd and 10 at a team’s own 20. This difference isn’t captured in rushing yards, but it plays a major role in my numbers. The Protrade win probability measure which similarly takes account of the game situation, also has this simultaneity problem.

Also on this issue of the simultaneity problem, there is at least one case in which a player’s contribution to his team’s chance of winning can be measured more cleanly, unexpected injuries and suspensions. For example, when Kevin Garnett was suspended for one game after throwing a punch (click here for the story), the drop in the odds of his team winning provides a nice measure of his impact.

(2) The Anticipation Problem. Certainly, the market builds in estimates of a team’s and players’ abilities. After all, the market gave the Colts a 68% chance of winning before they even stepped on the field. Certainly, this number says something about the Colts performance that isn’t captured in my net probability points statistic for each player. If the market expects that the Payton-Manning-led Colts offense will move the ball easily, then this expectation will be built into prices. As a result, Peyton Manning’s performance will indeed be understated by my measure relative to the performance of, for example, backup Jim Sorgi had he done the exact same thing as Peyton. However, it’s not so clear that Peyton’s performance will be understated relative to the performance of the Colts other offensive players. The anticipation problem holds for them as well. If the Colts offense is expected to move the ball easily, then receiver Reggie Wayne’s performance will also be understated relative to his performance without these high expectations.

Interestingly, if the market anticipated that the Bears wouldn’t have much success in moving the ball on the Colts, then this anticipation problem actually understates how bad Rex Grossman’s performance actually was! Despite the market’s low expectations of how the Bears would fair against the Colts, Rex contributed 36.5% (percentage points) to the Colts chance of winning by my measure, more than twice as much as any Colt. Just imagine how bad this number would have looked had the Bears actually been expected to beat the Colts.

There’s also a simultaneity problem with interpreting the market’s anticipation of the future as a pure problem. The fact that the market price incorporates expectations of the future can actually be an advantage for my measure. Just as the market price incorporates expectations of Peyton’s offensive ability, it also incorporates expectations of how bad the defense he’s playing against is. If the market expects the Colts offense to have success in part because the Bears’ pass defense is so poor, then my measure more accurately measures his performance than a measure that doesn’t take into account expectations of the Bears’ defensive ability. Certainly, a measure of a player’s performance should incorporate expectations of the defense’s ability. The Protrade’s current publicly available win probability measure doesn’t take expectations about particular teams’ defensive abilities into account, thus it will overstate the performance of offensive players when playing against a particularly poor defense.

I am big fan of Protrade’s attempts to measure a player’s performance in terms of his impact on the probability of his team’s chance of winning. However, estimates that only take into account historical data, and not market expectations, are bound to be inaccurate. Making estimates only taking into account historical data is like driving while looking in the rear-view mirror. There’s no problem as long as the road ahead is just like the road gone by. However, if the future isn’t like the past, then wham. (analogy courtesy Robert Merton). For example, just think about how the various new offensive strategies (recently, the spread in college football) can change the game. The market has an amazing ability to take into account all the particulars of the situation in a way that historical data cannot. Thus, I think probability estimates must take into account market expectations to measure accurately the probability impact of a play, which is essential to both of our measures of a player’s performance. The research director at Protrade assures me that they are working internally on a measure that takes team ability into account. I hope that their methods will be made public so that interested folks can see and test for themselves the accuracy of their measure.

In closing, I’m happy to see these probability based statistics getting more attention because I think they provide an exciting way to look at the game. Tom Brady provides an excellent example of the potential of these statistics. It’s a common notion that Tom Brady is better than his traditional statistics indicate because he has produced for his team in the moments when it mattered most. I’d love to see what a market-based probability performance measure would have looked like for Tom Brady in his many magical playoff performances and in his most recent dud.

SENTENCE OF THE DAY — Keith Jacks Gamble Edition — (Analogy from Robert Merton [*]):

Making estimates only taking into account historical data [and omitting market expectations] is like driving while looking in the rear-view mirror.

[*] Who the hell is Robert Merton????