Tradesports.com’s market for which bubble teams would make it to the NCAA Men’s Basketball Tournament mis-predicted three selections. This performance ties for last among the selections of thirty experts. The top performer was The Bracket Project, which wrongly predicted only one team, Syracuse instead of Arkansas. The consensus opinion of the thirty experts wrongly selected two teams, selecting Syracuse and Drexel instead of Stanford and Arkansas. Perhaps the most followed bracketologist, Joe Lunardi of ESPN also wrongly selected the same two teams as the consensus opinion. Tradesports.com’s market gave a higher probability of being selected to Syracuse, Drexel, and Kansas State in comparison to the selected teams Illinois, Old Dominion, and Arkansas. None of the thirty expert selections missed more than three teams. The chart below lists prices for bubble teams on Tradesports as of 5:45pm, just 15 minutes before the selection show. Of course, this sample is way too small to make any general conclusions about the accuracy of markets versus experts, but score one for the experts. 
People. Look at the volume.
My philosophy with regards to validity of PM predictions is: if an outcome trades under 500 contracts, it is not credible. If it’s 500-1000, it’s arguably credible, depending on how skeptically you perceive markets. If it’s over 1000, it’s definitely credible.
These markets are practically frozen. Volumes of 38, 31, 107 are essentially not trading.
You could apply the same reasoning to spread. Those spreads are enormous.
I practically went hoarse criticizing PMs’ performance for the 2006 elections; volume was quite sufficient. But I do not think this makes for a fair example.
Is this a designed experiment meant to yield a statistically significant result?
Doesn’t look that way. http://en.wikipedia.org/wiki/Design_of_experiments
But I only got a bachelor’s degree in applied math. Surely most of the commenters here will be smarter than me!
Alex, I agree that volume on these contracts is low. I also agree that more active markets make for more accurate predictions. However, I wouldn’t call this market frozen. Unlike election markets which are open for months and months before the event, this market only existed for one week. How does your philosophy with regards to the validity of PM predictions take in to account how long a market is open for trading? Also, the volumes of 38 and 31 that you point out aren’t for teams that the market wrongly predicted. All contracts in question have trading volume over 100.
I agree that the spreads are high, but I don’t think that they are too high to make the bid/ask median irrelevant. In only one case does the offer price for a wrongly-predicted-to-be-out team (Old Dominion 54) exceed the bid price for a wrongly-predicted-to-be-in team (Kansas St 53).
Caveat, my comparison of the market’s predictions to experts’ in this case is in no way capable of making meaningful statements of statistical significance. The fact that most experts outperformed the market in this case could be very well be a result of luck and not a general truth. In fact, I’m a believer in the power of markets to aggregate information consistently better than any expert or averaging of experts.
However, just because an observation doesn’t yield any results of statistical significance doesn’t mean that it should be ignored or never considered. Rather, it just means that there needs to be more observations before general conclusions can be stated with some confidence.
Well, said Mister Keith Jacks Gamble.
((((I remember that professor Koleman Strumpf said he would look for a spread around one dollar.)))
Say 100 contracts a day, on average. Although a lot of them are much more prone to spikes around a given time, and long periods of dormancy.
But you get the idea.
The low volume indicates a higher probability of a poor distribution of bias and thus inefficiency.
Pennock (and friends) have (repeatedly) shown that wisdom (and better decision making) can be derived with relatively little volume.
One has to look at the prices themselves, not just how they get ranked. Every expert got Syracuse wrong. For the rest of the mistakes, the markets called those essentially tossups indicating, rather correctly, that predicting the true bubble teams is nearly impossible.
Hi Lance, I agree that there is information in the market prices that I’m not using in my comparison. Unfortunately the experts don’t state probabilities, so the only relevant way to make the comparison in this case is the ranking. Certainly, the experts realize that their last few selections aren’t 100% locks to be in. I am assuming that the experts are choosing what they think are the most likely teams to get in. This assumption enables a valid comparison. I would love for the experts to state probabilities (or some sort of confidence indicator) along with their picks since this information would make for a much richer comparison.
As much as I’d like to say that the market’s existence makes all these experts on bubble teams superfluous, at least for this year, the evidence doesn’t support this view.
Seems to me this could also be an example of an uninformed crowd. American traders are barred from all TradeSports markets including the NCAA ones, right? So all the traders would have to be non-US. How many people outside the States follow NCAA college basketball anyway?
“American traders are barred from all TradeSports markets”
Most of them find it difficult to fund their TradeSports account now, but many hard-core US speculators remain faithful to TradeSports and managed to find ways to fund their account.