-

Philip E. Tetlock et al.:

Prediction markets can harness the “wisdom of crowds” to solve problems, develop products, and make forecasts. These systems typically treat collective intelligence as a commodity to be mined, not a resource that can be grown and improved. [*] That’s about to change.

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict major trends and events around the world over the next four years.

The Good Judgment Team, based in the University of Pennsylvania and the University of California Berkeley, will be one of the five teams competing — and we’d like you to consider joining our team as a forecaster. If you’re willing to experiment with ways to improve your forecasting ability and if being part of cutting-edge scientific research appeals to you, then we want your help.

We can promise you the chance to: (1) learn about yourself (your skill in predicting and your skill in becoming more accurate over time as you learn from feedback and/or special training exercises); (2) contribute to cutting-edge scientific work on both individual-level factors that promote or inhibit accuracy and group- or team-level factors that contribute to accuracy; and (3) help us distinguish better from worse approaches to generating forecasts of importance to national security, global affairs, and economics.

[*] –> ???

–> To sign up, go to goodjudgment.info.

Despite its importance in modern life, forecasting remains (ironically) unpredictable. Who is a good forecaster? How do you make people better forecasters? Are there processes or technologies that can improve the ability of governments, companies, and other institutions to perceive and act on trends and threats? Nobody really knows.

The goal of the Good Judgment Project is to answer these questions. We will systematically compare the effectiveness of different training methods (general education, probabilistic-reasoning training, divergent-thinking training) and forecasting tools (low- and high-information opinion-polls, prediction market, and process-focused tools) in accurately forecasting future events. We also will investigate how different combinations of training and forecasting work together. Finally, we will explore how to more effectively communicate forecasts in ways that avoid overwhelming audiences with technical detail or oversimplifying difficult decisions.

Over the course of each year, forecasters will have an opportunity to respond to 100 questions, each requiring a separate prediction, such as “How many countries in the Euro zone will default on bonds in 2011?” or “Will Southern Sudan become an independent country in 2011?” Researchers from the Good Judgment Project will look for the best ways to combine these individual forecasts to yield the most accurate ‘collective wisdom’ results. Participants also will receive feedback on their individual results.

All training and forecasting will be done online. Forecasters’ identities will not be made public; however, successful forecasters will have the option to publicize their own track records.

Who We Are

The Good Judgment research team is based in the University of Pennsylvania and the University of California Berkeley. The project is led by psychologists Philip Tetlock, author of the award-winning Expert Political Judgment, Barbara Mellers, an expert on judgment and decision-making, and Don Moore, an expert on overconfidence. Other team members are experts in psychology, economics, statistics, interface design, futures, and computer science.

We are one of five teams competing in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Agency). The ACE Program aims “to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts.” The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public.

-

More at their blog.

—

-

UPDATE: GMU team.

Joke aside, all that is just aesthetics. It remains that:

1. Collective forecasting brings an added accuracy that is microscopic —if any.
2. Collective forecasting brings objectivity —which people but us don’t value.
3. It takes time to perform the information aggregation, which disqualifies collective forecasting as a useful tool.

So, inventing a new term does not mean that you create a new reality, doc. The reality is that Nate Silver does not give the first fig about collective forecasting. Nate Silver is the king of forecasting (and so is his buddy Andrew Gelman) —not InTrade, not BetFair, not Robin Hanson, not Justin Wolfers, not David Pennock, etc.

forecast (v.)
c.1388, “to scheme,” from fore “before” + casten “contrive.” Meaning “predict events” first attested 1494.

Previously: Apple dictionary on “predicting” vs “forecasting”

Previously: Andrew Gelman on “predicting” vs “forecasting”

-

THE END RESULT OF THE MEMORIAL DAY FIGHTING OVER TERMINOLOGY:

1. Etymology indicates that the word “forecasting” is OK to use as a meaning for “predicting”.
2. Predictive modeling expert Andrew Gelman can’t see any difference between the 2 words, and that guy is an expert —Robin Hanson is not, and I am not.
3. It is true (as I said from day one) that the word “forecasting” is treated differently than the word “predicting”in the dictionaries. If Robin Hanson had a good pair of eyeglasses, and were humble and modest enough to read dictionaries, he would see that, too.
4. However, point #1 and point #2 are more important to moi. (See, abiding forever by point #3 means that the recent past should always govern us. Obviously, wrong. We should free ourselves from the chain of the past and re-invent things, when necessary.)
5. Hence, I declare the words “forecasting” and “predicting” as perfectly equivalent —a complete U-turn [*] that will give credence to the untrue rumor that I am “bi-polar”.
6. And so, our good doctor Robin Hanson wins the fight (and I lose miserably, the face planted in the dirt, while the prof tramples my corpse with a joyful vengeance), and he is allowed to use the term “collective forecasting” to mean “collective intelligence that predicts”.
7. But the discussion is not over (so as to piss off our good friend Mike Linksvayer a bit further). Is there a better term than “collective forecasting”?…

[*] My category “forecasting” is all messed up, now. I used to file there things about predictive modeling and polling, exclusively.

QUESTION (from LinkedIn): Forecasting the same as prediction? Which one is more realistic and easier to do?

ANSWER: Forecasting is different from predicting. Predicting is much easier but far less accurate.

Predicting is when you start making guesses about things. For example, you predict that laying sheet-rock will take 45 hours to do and you guess that it will be done in 2 weeks.

Forecasting, on the other hand, is when you take information from past jobs and apply it to a new job. For example, if you have seen that laying the sheet-rock for a 3,000 sq ft space takes 65 hours and it usually done in 4 weeks then the next time you have to quote out the same job you’ll be able to forecast how much its going to cost and how long it will really take i.e. when it will really be done after work starts on it.

The big difference is predicting is based on your best guess from experience. Forecasting is based on data you’ve actually recorded and tracked from previous jobs.

Previously: Apple dictionary on “predicting” vs “forecasting”

Previously: Andrew Gelman on “predicting” vs “forecasting”

NEXT: Predicting = Forecasting –> Collective Forecasting = Collective Intelligence That Predicts

I suspect the words have different meanings in different contexts.  In statistics, “prediction” is often used even when the result has already happened:  that is, if you have a model, y = f(x) + error, then f(x) is said to be the “predicted” value.  So you can have a predicted value, even for an observation you’ve seen.  I’ve worked in an example using climate models where the term “hindcast” was used, because we were fitting a model from 40 years of data and then using it to predict values that had already occurred.

In common usage, I think “prediction” refers to predictions of the future. Consider the famous Niels Bohr quote, “Prediction is very difficult, especially about the future.”  I assume he as being ironic and was assuming that prediction is always about the future.  I can only assume forecasting is also about the future, given the “fore” at the beginning of the word.  So I don’t really know the difference between prediction and forecasting.

I’ve never heard the term “collective forecasting,” but I’m sure that reflects my ignorance rather than anything else.

Previously: Apple dictionary on “predicting” vs “forecasting”

Andrew Gelman’s website

Andrew Gelman’s excellent blog

Andrew Gelman sometimes guest-blog at Nate Silver’s blog.

NEXT: Predicting = Forecasting –> Collective Forecasting = Collective Intelligence That Predicts

THE RIGHT WORD

While all of these words refer to telling something before it happens, predict is the most commonly used and applies to the widest variety of situations. It can mean anything from hazarding a guess (: they predicted he’d never survive the year) to making an astute inference based on facts or statistical evidence (: predict that the Republicans would win the election).

When a meteorologist tells us whether it will rain or snow tomorrow, he or she is said to forecast the weather [*], a word that means predict but is used particularly in the context of weather and other phenomena that cannot be predicted easily by the general public (: statistics forecast an influx of women into the labor force).

Divine and foreshadow mean to suggest the future rather than to predict it, especially by giving or evaluating subtle hints or clues. To divine something is to perceive it through intuition or insight (: to divine in the current economic situation the disaster that lay ahead), while foreshadow can apply to anyone or anything that gives an indication of what is to come (: her abrupt departure that night foreshadowed the breakdown in their relationship).

Foretell, like foreshadow, can refer to the clue rather than the person who gives it and is often used in reference to the past (: evidence that foretold the young girl’s violent end).
Augur means to foreshadow a favorable or unfavorable outcome for something (: the turnout on opening night augured well for the play’s success).

Prophesy connotes either inspired or mystical knowledge of the future and suggests more authoritative wisdom than augur (: a baseball fan for decades, he prophesied the young batter’s rise to stardom).

Although anyone who has inside information or knowledge of signs and symptoms can prognosticate, it is usually a doctor who does so by looking at the symptoms of a disease to predict its future outcome.

[*] So much for the 3 commenters (Paul, Daniel, and Mike) who didn’t get my point about weather forecasting being the most understood form of forecasting —one that is based on predictive modeling, WHICH THE PREDICTION MARKETS (AND THE OTHER NON-TRADING MECHANISMS THAT PREDICT) HAVE NOTHING TO DO WITH.

NEXT: Andrew Gelman on “predicting” vs “forecasting”

NEXT: Predicting = Forecasting –> Collective Forecasting = Collective Intelligence That Predicts

“Collective prediction” sounds a lot more awkward to my ear than “collective forecasting.” “Collaborative prediction” sounds just as bad. I don’t think most people see much a difference between “forecasting” and “prediction”, other than that, “forecasting” is a word used a lot more often in business contexts. That seems to me a good reason to use “forecasting.”

I am waiting an input from Andrew Gelman —maybe.

By the way, Felix Salmon finds out that I am indeed bi-polar… — .

NEXT: Predicting = Forecasting –> Collective Forecasting = Collective Intelligence That Predicts

1. I wrote that 2 EPM software providers (NewsFutures and CrowdCast, representing half of this industry) are turning their back from the trading technology and are opting for the non-trading technologies.
2. The NewsFutures CEO insisted to let the world know that, oh no, no, no, NewsFutures haven’t abandoned the trading technology, and that they are proud to be able to propose both to their customers.
3. That was a PR point he wanted to make (so as to sound that NewsFutures is the omni-solution provider, wow, you should be so impressed). But the piece of truth that our readers should have in mind is what I am telling them: NewsFutures and CrowdCast don’t believe anymore that EPMs are the right solution to propose to companies.
4. While you can have a very interesting conversation, privately, with Emile on that, all of the sudden, when a question is publicly asked to His Majesty, then the PR machine is all silent.

The question was:

NewsFutures does list here 4 tools. Could Emile tell us how popular each of these tools is? I would like a percentage for each of these 4 tools. In other words, could Emile quantify EPMs in the whole collective intelligence solution panorama? Is that inferior to 50% or superior to 50%? Please. Thanks.

- What do you make of the term “collective forecasting”, coined (apparently) by our good doctor Hanson? Do you think it is appropriate? Does it convey the right message about the specificity of our information aggregation mechanisms that predict (our prediction markets and the non-trading mechanisms)? Or, at the contrary, does it confuse people into thinking about data and models (as in “weather forecasting“), which is obviously something we are *not* into?

- Should we let Robin Hanson use the term “collective forecasting”, or should we stop him cold right now before more damages are done? Let us know. Please, do comment. We need your input. Mike, David, Jason, Keith, Daniel, etc. Don’t hesitate to be libelous. Be harsh. Be disrespectful. Tell the truth.