I’m glad to hear that the soccer markets did become efficient, eliminating the home team bias. Perhaps it takes time for bettors to understand the market and learn how to take advantage of profit opportunities. While we have had stock markets for centuries, prediction markets are relatively young.

Your review of these betting markets and their accuracy is encouraging.

I didn’t explain myself very well. I used this as an example.
There was a home-country bias once, a very long time ago when betfair first started. I only heard stories about this. These days the soccer markets are very efficient and they have been for many years. Probably because of the factors you mentioned – huge liquidity, huge potential for profit, a lot of trader diversity applying all sorts of startegies, betfair offering valuable data for free, etc. etc.
A couple of years ago I must have studied at least 30 different factors like this one in an attempt to gain a systematic edge. I dug deep. I only found 4 or so factors giving me an edge but wasn’t able to beat commission after three months of systematic betting. That’s how efficient they are, or were, even 3 years ago. I don’t have a problem saying that these markets are accurate because of my findings.

But what we’re looking at, in most cases, when we study prediction markets, is really, to a large extent inefficient markets. Because they are so fresh. Therefore they cannot have reached their full potential with regard to accuracy. How to get there we’ve discussed and you (and I) have mentioned a couple things we can do and need to do before we can make a final judgment on their suitability.

Yes, we need a LOT more research. So much perhaps that it is almost impractical. And I am very skeptical of the results that will come out of this. Well, we’ve done our share in pointing out some important issues.

You are very correct to say that it is very important to know the degree of over/under estimation error in these markets. If you know the market overestimates the outcome, you can virtually eliminate the possibility of the outcome being an even higher percentage (in your football markets). That is, using your example, the actual outcome is most likely to be 75% or less, but rarely higher than 75%.

Since you have found that these markets consistently overestimate England’s winning percentage, the market is not “efficient”. Perhaps there’s a home-country bias? If you make this finding public (as you have), presumably the market will become more efficient as more traders try to profit from the bias. That’s what we would expect to occur. Hope you have already made your profit!

As to whether an efficient market also means that it is an accurate one, it is not necessarily so. As we saw in the movie markets, they were relatively efficient in the sense that the markets did accurately reflect the initial prior information of the traders. They were not accurate with respect to the outcomes, however. My explanation is that the aggregate information held by the pool of traders was not complete. If the trader pool (as a whole) does not possess most of the information necessary to estimate the outcome, they won’t be able to create the missing information by trading, and the market is unlikely to be accurate (and it shouldn’t be).

If a market is as good as it can be and it is still not accurate enough for decision-making purposes, perhaps this is an issue that is not suitable for prediction markets. Alternatively, you need to improve the inputs to the process – more traders, more diversity, more complete information, fewer imperfections in the market, more liquidity, etc…

More research needs to be done on the factors that help improve accuracy. Efficiency is great, but it will never replace accuracy in the business world.

When I say the market consistently overestimates (or better, has a tendency to) overestimate or underestimate the result (I mean the outcome, here) then it follows that the market cannot be accurate. I’m looking at 100 games played by the English soccer team on 100 seperate markets and the market predictions (as reflected by the average price per market) are telling us that the English team has an average chance of 75% of winning when the actual outcomes tell us it is only 65%. Hence, the market is overestimating the chance of England winning.

In this case, the market is also not efficient. I look at it from a business perspective. It is not efficient because a trader betting on England every time will lose a lot of money while someone betting against England will make money (even when taking commission into account).
Betting exchanges don’t want this because the loser might walk away. They want markets to be as efficient and accurate as possible. That’s why you can download all that data for free on betfair’s site and analyze it, if you are a customer.

To what degree markets are overestimating/underestimating the actual result is very important when we’re considering market accuracy. More important than most people think, that’s why I keep coming back to this. And we can look at many different factors influencing the outcome and analyze whether the market predictions reflect this accurately (no overestimation and no underestimation). This is very important.

When a market IS efficient does that mean it is also accurate? Now it becomes interesting. First of all, that’s a subjective matter. How accurate do you want it to be? How accurate do you need it to be? This can all be calculated when you have enough data (and hopefully it will be more than 7 markets!). What’s important to know is, that the market is as accurate as it can be, or at least, that there isn’t a whole lot you can do to improve it. Most of the rest of it is probably noise.
You see, I’m sort of reverse engineering your quest for accuracy.

I agreed with the conclusion that private information was being reflected in the market prices and distributions, but I questioned the accuracy in these particular markets. Basically, I’m saying that the markets work, *but* they were reflecting the information held by the traders, which was incomplete. That is, the traders, collectively, did not possess enough information to accurately predict the outcome. In this situation, prediction markets will *never* be able to create the missing information in order to arrive at an accurate prediction.

You could measure the standard deviation of the last 100 movie receipts, but it would be the deviation around the mean (average) of the movie receipts, not the predictions of the future movie receipts. These are two very different things. Also, “movies” are not homogeneous. The average box-office receipts and its associated standard deviation are kind of meaningless when trying to forecast receipts for a particular movie. These prediction markets are trying to be much better than averaging historical information.

With respect to the predictions, they are the means (averages) of the traders’ forecasts. The authors of the paper assumed a normal distribution, which allows for the use of the standard deviation as a measure of dispersion of the trades. (I doubt the market trades follow a normal distribution, but it is irrelevant for my analysis). Half of the trader forecasts will be above the market prediction. I assume this is what you mean regarding “the market consistently overestimates/underestimates the result.” Here, “result” must be the market prediction. In the paper, the market predictions were not consistently above or below the actual outcomes.

I understand your argument that the markets might not have been efficient, but they did operate for 4-14 days prior to the movie openings, and the time period for trading did not appear to affect the ability of the markets to reflect the traders’ information. Further, the length of trading did not affect the accuracy of the predictions. The fact that the market were able to reasonably accurately reflect the private information of the traders (and the dispersion) indicates that the market did reach some sort of equilibrium.

Given that the market predictions were not particularly accurate, nor were they consistently so, indicates that the trader pool did not possess the necessary information (collectively) to come up with the right prediction.

Perhaps these markets for movie receipts were just too difficult to predict, at least by these trader pools.

This research wasn’t supposed to measure accuracy, you are. And I don’t know who calculated that standard deviation or what purpose it should fulfill, but if I were to measure accuracy I would take the last 100 or so real-life box office receipts and get my SD from there. In this case your SD would probably be closer to $50M than $10M and most predictions (from your 7 markets) would probably fit within 1 SD.

>> “meaning that the actual outcome could be between $71M and $129M. Is this level of accuracy adequate for proper decision-making?”

That has become a subjective matter but I would argue yes if the true outcome could just as well be be $10,000 or $800,000,000.
But again, what matters is whether the market consistently overestimates/underestimates the result OR not. If it does, the market is not efficient. Experienced and knowledgable traders will be able to detect that and exploit it financially given the right conditions (discussed before). As a result the market will become more efficient and more accurate. These things will work out for themselves (including a tight distribution) given the right conditions. And I know we both agree on that.

I rounded. It is (7-5)/7= 28.57% – say 29%. The underestimated error was (3-5)/3 = -67%. You need to calculate the error using the *prediction* as the base, because that is the number you on which you would be basing decisions. I understand what you’re saying about the absolute value of the error, however, I believe this is incorrect. Here’s why.

In the study, some market predictions were above the actual outcome and some were below. There was no way to tell *beforehand* whether the prediction would be too low or too high. The predictions were the *means* for each market. The distributions (dispersions) were around these means, *not* around the actual outcome. Had the dispersion been around the true outcome, then it wouldn’t matter whether the estimates were below or above the actual outcome, as long as they were close enough. But this doesn’t make much sense, because there is no distribution of participant trades around the *actual* outcome. The distribution is around the mean of their estimates, reflected in their trades. Therefore, it is very important to know whether the markets are under- or over-estimating the actual outcome. The decision-maker does not make decision based on the actual outcome, it is made based on the forecast or predicted outcome.

For example, let’s say there’s a prediction market estimate of $100M for Movie A’s gross receipts. From the trading data, we can determine the standard deviation. Let’s say it is $10M. Therefore, we can say that we expect that 68% of the time, Movie A’s gross receipts will fall within the range $90M – $110M. The prediction market will be well calibrated, if this holds true over a series of similar markets. Now let’s look at it from the studio executive’s viewpoint. In these studies, the average absolute error was calculated at 29%, meaning that the actual outcome could be between $71M and $129M. Is this level of accuracy adequate for proper decision-making?

The noise and uncertainty is basically an estimate of the information that is *lacking* in the market. It is reflected in the dispersion of market prices around the mean. There will always be some price dispersion, caused by unknown factors, “noise” bettors, traders with different search costs, etc… The trick is to develop markets that make predictions with relatively tight distributions. Of course, the prediction must be reasonably close to the actual outcome, *too*.

you lost me again. How could they possibly understate the error since we are talking about absolute values. Maybe you can give a simple example and show me how that 29% was calculated.

As for my 3,3,7,7 example, they could be very accurate predictions if the pool of possible (and likely) outcomes is between -100 and +100, for instance.
A horse could be trading at 100 to 1 or 2 to 1 in which case a 29% error rate isn’t that bad (as you could easily be off 400%). The standard deviation should help us make that assessment but in your case it isn’t clear how that was calculated. And I’m pretty sure they didn’t calculate is the proper way.
Also, the amount of noise or uncertainty is important. Take a soccer match and you will notice that there are a lot of unpredictable factors which make it hard to predict a result. Luck being one of them. You could easily get 4 out of 4 “incorrect” but that doesn’t mean that the predictions were inaccurate. I can assure you that they are.
What’s important to know is that these markets overstate the result 50% of the time, and understate them 50% of the time. You can study like 20 different factors like weather conditions, home/away etc. and you will get the exact same result every time – 50/50. This is what accuracy is primarily about, to get the best possible estimate. Nothing more, but …VERY useful to EPM.

We agree on many things, and one of them is that it is very unfortunate that the article didn’t consider any of this. This is not unusual, you will see many math whiz kids come by on the forums on betfair, and their approach usually sucks.
So far you have impressed me so don’t get me wrong. But something is wrong about the way you interpret the data even though it is hard for me to judge because I didn’t read the article.

I’ll try to explain this a bit better than I did. I used the word “disguising” not to imply anything underhanded, but to highlight that the authors of the study failed to consider the errors went both ways. The problem, in this case, was that we didn’t know whether the error was under- or over-stating the actual outcome. There was no consistency from one market to the other. Therefore, by using the absolute value of the percentage error, they were understating the error by as much as 100% (it could be a 29% error on either side of the mean prediction). Not really accurate, in any useful sense of the word.

I’m not sure what you mean with your numerical examples. If you were running four markets that provided predictions of 3, 3, 7 & 7, you cannot argue that any of them are accurate, if the actual outcome should be 5. The first two have an error of -67% and the other two have errors of +29%. The problem is we don’t know *beforehand* which way the error is going to go. For prediction markets to be useful, these errors have to be reduced, substantially.

Paul

What matters is that prediction markets are accurate ON AVERAGE. This way you won’t be able to make money in the market, because you won’t be able to devise a system, because the determining facors causing the discrepancy will likely be noise. There will ALWAYS be noise. I will even go so far as saying that the amount of noise is irrelevant to the accuracy of predictions markets. If there is a lot of noise (when there are a lot of random influencing factors, which you can’t predict), then you won’t be able to change that, EVER.

So, even these results
3,7,3,7
could point to prediction markets being fairly accurate. It depends.

In other words, a market prediction will always be an estimate. What matters is whether the market has a tendency to overestimates or underestimates the result when considering multiple variables/factors. Of course, how much standard deviations a prediction is from the mean is also important, but that is not my point.

Although I think Paul is trying to point that out (not sure) his choice of words with regard to “disguising” is rather unfortunate.

In this context, I also don’t understand where this is coming from or what he is trying to say.
Quote:
“Using the absolute percentage error disguises the fact that the errors go both ways (some were under- and others were over-estimated). Further, the prediction markets provide no guidance as to which way the error is likely to fall. Therefore, the real error is much larger than the absolute (value) of the percentage error. It is, perhaps, as much as twice the error calculated by the authors. Consequently, the real prediction market error might be as high as 58% in these markets.”