Should we believe traders who make the most money, relative to traders with good statistical forecasting scores? – NO.
Yet another PM post over there, at Overcoming Errors dot com (where the main error is to focus on bias and not on errors). In passing, note that the Robin Hanson’s blog posts are often published at 06:00 AM —a time where most honest people are still in their bed. The guy is probably making use of an automatic TypePad function (so as to pretend that he is already up and shaved at that early hour –even on a Sunday). I should inquire whether Word Press (the open-source blogging software used for Midas Oracle) has the same kind of function —keeping up with the Jones, you know.
—
Does Profit Rate Insight Best? – [trader's profits vs. trader's predictive power] – by Robin Hanson – 2006-12-03
Upcoming in Theory and Decision (in the same issue as I), David Johnstone asks a fundamental prediction market question: should we believe traders who make the most money, relative to traders with good statistical forecasting scores? His answer is “No”:
A gambler who wants to maximize future profits should trade on the advice of the analyst cum probability forecaster who records the best probability score, rather than the highest trading profits.
The reason is simple: when a forecaster’s prediction falls within a bid-ask trading spread, he cannot improve his trading profit, but he can still improve his probability score. The extreme case is even simpler: if no market exists, he can make no trades, but he can still improve his score.
Another reason to prefer a statistical score is that the trader’s goal might be something other than maximizing the logarithm of their final trading profit. Only for log utility traders with no bid-ask spread could profit reveal as much as a log scoring rule.
On the other hand, if there is not just one statistical score to use, then a forecaster might have selected the score to show you that made them look best, reducing its info value. This is much harder to do with trading profits.
Finally, let me note that my automated market maker, which I call a “market scoring rule,” has a zero bid ask spread. So profit trading with it will rate someone just as well as a statistical score, at least if a combinatorial version is used that allows for all the combinations that a scoring system might allow.
—
Technical Note: Go to his blog post. Put the cursor of your mouse on each of his three links. Each time, let the cursor idle on the link for at least one good second or two. What do you see? NOTHING HAPPENS. Now put your same cursor on any link that you see above, in the text that I have treated. What do you see? You see a tiny pop-up message, which is called a “link title”. This link title is very important for the reader: it tells him/her more information about the webpage he/she will land on if he/she clicks on that link. (A second inkling comes from the status bar, at the bottom of the browser window, which spells out the URL of that link.)
Either Robin Hanson types his text in a HTML editor that doesn’t do link title (and he later pastes the code source in TypePad), or his proprietary blog editor, TypePad, does not support link title. Or it could be that TypePad does support link title, but our lazy doctor doesn’t bother filling the “link title” box. In all cases, it’s bad practice.
Required Reading For Our Good Doctor Robin Hanson:
- Using Link Titles to Help Users Predict Where They Are Going – by Jakob Nielsen – 1998-01-11
—
Technical Note #2: Here’s more advice from Jakob Nielsen —PUT IN BOLD YOUR MOST IMPORTANT SENTENCE(S). Funny enough, it’s an important principle which our good doctor Robin Hanson is totally refractory to —for a totally irrational and mysterious reason. (Remember: that’s a guy who has set up a group blog on rationality, and how to overcome bias. If he asked me, I’d have some suggestion on where to start. What about: how to overcome the anti-Internet usability bias?)
- Microcontent: How to Write Headlines, Page Titles, and Subject Lines – by Jakob Nielsen – 1998-09-06
- How Users Read on the Web – (They don’t.) – by Jakob Nielsen – 1997-10-01
- Writing for the Web – by Jakob Nielsen – often updated
