[This article by a guest author has been contested, for the reason that information about the bigger picture was missing. It is retired.]

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

I had a question about VentureBeat’s statement, and Leslie Fine has answered it.

VentureBeat:

As everyone has known since the modern corporation was invented, employees such as executive assistants or production line managers often have better information and more accurate insight than their management.

CrowdCast’s Leslie Fine:

The claim here is not that the administrator knows more than the executive.  The claim is that the executive’s point-of-view can be enhanced by mingling it with the admin’s. Empirical and experimental evidence show that in most cases the team (including the best predictors) outshines the best predictor working alone. Furthermore, in our implementations we find that our front line managers often have earlier information than the executive suite.  Our mission is not only to provide accurate forecasts, but to do so as early as possible.

CrowdCast’s website

Best wishes to CrowdCast. They have a good Chief Scientist.

Previously: CrowdCast = Collective Forecasting = Collective Intelligence That Predicts

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

At VOX, David Hendry and James Reade examine the question, “How should we make economic forecasts?” Among the ideas discussed is whether prediction markets could be used to improve economic forecasting. Interesting suggestion and seeming to be worthy of additional exploration, but the authors don’t go too deep here.  Instead, they assert that “prediction markets can be viewed as a form of … model averaging,” and then drift into a discussion of forecast averaging. I’m not sure that forecast averaging is a good way to look at prediction markets.

Here is what they say:

Prediction markets can be viewed as a form of forecast pooling or model averaging, a common forecast technique (Bates and Granger 1969, Hoeting et al 1999 and Stock and Watson 2004). That is, forecasts from different models are combined to produce a single forecast. In prediction markets, each market participant makes a forecast based on his or her own forecasting model, and the market price is some function of each of these individual forecasts.

Since the “prediction” implied by a prediction market is set by the marginal transaction, it depends not at all on the distribution of earlier trades, nor on the valuations of parties priced out of the market at the current price.

For example, consider two event markets: in the first 999 contracts trade at $0.50 and the 1000th and final trade is at $0.75; in the second 999 contracts trade at $0.76 and the 1000th and final trade is at $0.75.  In the typical interpretation of prediction markets, the event is “predicted” to result with a 75 percent probability in both cases.  However, averaging among the different predictions doesn’t get you that result.

(Well, strictly speaking the market price is “some function” of the prices – namely, one in which all trades but the last are weighted zero and the last trade is weighted one. You can call this “averaging,” but that isn’t the most useful explanation of the function.)

I’m not arguing that forecast averaging might not be a good idea in many situations, just that averaging doesn’t seem like a good way to explain what a prediction market is doing.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

- A Lesson in Prediction Markets from the Game of Craps – by Paul Hewitt

- Why Public Prediction Markets Fail – by Paul Hewitt

Both articles are required reading for Jed Christiansen and Panos Ipeirotis (alias “Prof Panos”).

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

Niall O’Connor:

Robin Hanson, is what I would term a market fundamentalist – somebody who believes that markets are the primary mechanism for achieving the public good. Recent events have demonstrated that this creed is morally bankrupt. We are, in the words of Michael Sandel, “at the end of an era of market triumphalism”.

Robin Hanson:

I deny the accusation, though I’m not sure why I should have to. Having a high opinion of the eventual info power of decision markets [*] is very different from saying “markets are the primary mechanism for achieving the public good.” Shouldn’t you have to first provide support for your accusation, rather than me somehow having to first prove bald accusations wrong?

[*] That’s at the core of the issue. Somebody who has a “high opinion” of something that has not given stellar results since 1988 is obviously exaggerating. I personally have a somewhat good opinion of prediction markets and conditional prediction markets, but not a “high opinion” —sense my nuance. Niall O’Connor has been too strong in his “accusation” ( ), but he is a bit in the right. [UPDATE: Niall O'Connor has an honest reply to Robin Hanson.]

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

Panos Ipeirotis hates it when I report an individual prediction market “failure“. (”Stupidity” is how he labelled it. )

1. I am a fan of Panos Ipeirotis’s scientific approach on prediction markets. It is all right.
2. However, I (obviously) won’t use the statistical and probabilistic approach when reporting about the expiry of one (or two) prediction market(s) in particular.

Let me explain why.

1. Panos Ipeirotis is a professor whose job is to instruct young people about the scientific approach. Fine.
2. On Midas Oracle, I abide also by the marketing approach. That is, I try to understand how people view things. Sensing how people can understand or mis-understand prediction markets is what I try (hard) to do. Turn out when I talk about prediction markets to people, the first question they ask me is, “How did prediction markets fare with Barack Obama (or George W. Bush)?“.

My observations:

1. People don’t want to be fed with statistical series about multiple historical prediction markets.
2. People want to know whether one prediction market did or did not forecast the outcome of an event —in terms of yes/no, right/wrong, correct/false.
3. People treat a probabilistic prediction just like any binary prediction.

Now, let’s take a look at how Nate Silver did communicate with the public on The Oscars 2009:

1. Nate Silver gave a set of probabilistic predictions.
2. However, when he blogged his failure, he acknowledged it in the binary mode (yes/no, right/wrong, correct/false) —”four hits and two misses“.

That’s how I do it, too. When one prediction market expires, I assess its past prediction as if it were binary (and not probabilistic) —in terms of success/failure. Which makes me an idiot in Panos Ipeirotis’ book. – (I can take it.)

Now, my challenge to Panos Ipeirotis, is this:

1. Do blog the next expired prediction market (on any topic).
2. Do apply your scientific approach.
3. I bet that your blogging will be as boring as a fruit cake, because, by essence, you can’t say anything, scientifically, about one expired prediction market —or if you abide by the statistical and probabilistic approach, then you won’t really report on that expired prediction market.

Don’t get me wrong. Midas Oracle should publish scientific assessments about multiple historical prediction markets. But to report about one expired prediction market, and to have a bit of fun, it is necessary to put (temporarily) science on the back burner.

As an addendum, remember that Emile Servan-Schreiber told Robin Hanson that science is not all.

NEXT: The prediction markets chalk another one up, as Susan Boyle is sent packing.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

Paul Hewitt:

The point of this discussion is that prediction markets should be well-calibrated, but this is not a sufficient condition for their usefulness. They must also provide accurate predictions, with relatively tight distributions. The maximum allowable dispersion of the distribution will depend on the materiality of the forecast error. That is, the prediction should be accurate enough, such that the maximum allowable error would not cause the decision-maker to alter his or her decision had the true value been known in advance.

Where the distribution of the prediction market is not tight, the market may still have some use, but not so much for being able to predict the outcome. Instead, the market will be providing information about the degree of uncertainty surrounding the outcome. This may indicate the need for greater care in assessing risks and the need for more extensive contingency planning. A flatter distribution may indicate that the market is not functioning properly (lack of information completeness, perhaps). Alternatively, a flat distribution may indicate that the variable being predicted is, simply, not predictable.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

In honor of our good doctor Robin Hanson, I am creating a new category, “collective forecasting“ —and the “forecasting” category becomes “traditional forecasting“, to mark the difference. (Can you sense it, Paul Hewitt?) The prof should take this as a great honor. The last time I created a new category, that was “science“, in honor of Albert Einstein. So, Robin Hanson is in good company.

Joke aside, all that is just aesthetics. It remains that:

1. Collective forecasting brings an added accuracy that is microscopic —if any.
2. Collective forecasting brings objectivity —which people but us don’t value.
3. It takes time to perform the information aggregation, which disqualifies collective forecasting as a useful tool.

So, inventing a new term does not mean that you create a new reality, doc. The reality is that Nate Silver does not give the first fig about collective forecasting. Nate Silver is the king of forecasting (and so is his buddy Andrew Gelman) —not InTrade, not BetFair, not Robin Hanson, not Justin Wolfers, not David Pennock, etc.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

What is the etymology of the word forecasting?

forecast (v.)
c.1388, “to scheme,” from fore “before” + casten “contrive.” Meaning “predict events” first attested 1494.

Previously: Apple dictionary on “predicting” vs “forecasting”

Previously: Andrew Gelman on “predicting” vs “forecasting”

-

THE END RESULT OF THE MEMORIAL DAY FIGHTING OVER TERMINOLOGY:

1. Etymology indicates that the word “forecasting” is OK to use as a meaning for “predicting”.
2. Predictive modeling expert Andrew Gelman can’t see any difference between the 2 words, and that guy is an expert —Robin Hanson is not, and I am not.
3. It is true (as I said from day one) that the word “forecasting” is treated differently than the word “predicting”in the dictionaries. If Robin Hanson had a good pair of eyeglasses, and were humble and modest enough to read dictionaries, he would see that, too.
4. However, point #1 and point #2 are more important to moi. (See, abiding forever by point #3 means that the recent past should always govern us. Obviously, wrong. We should free ourselves from the chain of the past and re-invent things, when necessary.)
5. Hence, I declare the words “forecasting” and “predicting” as perfectly equivalent —a complete U-turn [*] that will give credence to the untrue rumor that I am “bi-polar”.
6. And so, our good doctor Robin Hanson wins the fight (and I lose miserably, the face planted in the dirt, while the prof tramples my corpse with a joyful vengeance), and he is allowed to use the term “collective forecasting” to mean “collective intelligence that predicts”.
7. But the discussion is not over (so as to piss off our good friend Mike Linksvayer a bit further). Is there a better term than “collective forecasting”?…

[*] My category “forecasting” is all messed up, now. I used to file there things about predictive modeling and polling, exclusively.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To

Robin Hanson:

There is now a prediction market industry, where I do a lot of consulting, most of which is to private companies.  Most of these are on prediction markets, but some are decision markets.

What Robin Hanson does not say to his gullible readers (mostly, young collegians who have no experience of real-life forecasting) is that:

1. When you ask any non-ideological researchers, these days, on the value of prediction markets over simpler collective intelligence mechanisms, they tell you either that the benefit is very, very small or that it is non measurable (i.e., it probably does not exist).
2. Half of the so-called EMP industry (a microscopic industry that lives on hype and trial balloons, it should be said), which Robin Hanson is so proud to belong to, has abandoned enterprise prediction markets. Notably, NewsFutures (founded by a Robin Hanson fanboy, and advised for years by Robin Hanson himself) and CrowdCast (a VC-backed venture) have (independently of each other) decided to stop selling proposing EPM SaaS only to customers, and focused instead on simpler and more social mechanisms. For more information, I recommend my readers to go conversing with Emile Servan-Schreiver of NewsFutures, who has a lot to say about this issue. Emile is well versed into both the research and the practical sides of collective intelligence used within companies. [UPDATE: See Emile's comment.] Mat Fogarty (who, him, as experience in real-life forecasting for business) and Leslie Fine (a scientist) of CrowdCast are also good interlocutors.

Contrary to Overcoing Whatever, Midas Oracle publishes both the good and the bad about prediction markets. We do not censor the bad news. We do not cherry-pick the good news. You will notice, in the near future, that we will publish information that supports the opposite thesis (that is, EPMs are the panacea), and then we will link to Inkling Markets as the best interlocutor in town. We try to publish about both sides (EPMs are no good, EPMs are good), not because we are “bi-polar”, but because the jury is still out on this matter, and our readers need to have both arguments put under their very nose. Act one today.

Midas Oracle = Archives + Best + Predictions + Links + Publish + How To