Philip Tetlock:

-

Philip E. Tetlock et al.:

Prediction markets can harness the “wisdom of crowds” to solve problems, develop products, and make forecasts. These systems typically treat collective intelligence as a commodity to be mined, not a resource that can be grown and improved. [*] That’s about to change.

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict major trends and events around the world over the next four years.

The Good Judgment Team, based in the University of Pennsylvania and the University of California Berkeley, will be one of the five teams competing — and we’d like you to consider joining our team as a forecaster. If you’re willing to experiment with ways to improve your forecasting ability and if being part of cutting-edge scientific research appeals to you, then we want your help.

We can promise you the chance to: (1) learn about yourself (your skill in predicting and your skill in becoming more accurate over time as you learn from feedback and/or special training exercises); (2) contribute to cutting-edge scientific work on both individual-level factors that promote or inhibit accuracy and group- or team-level factors that contribute to accuracy; and (3) help us distinguish better from worse approaches to generating forecasts of importance to national security, global affairs, and economics.

[*] –> ???

–> To sign up, go to goodjudgment.info.

Despite its importance in modern life, forecasting remains (ironically) unpredictable. Who is a good forecaster? How do you make people better forecasters? Are there processes or technologies that can improve the ability of governments, companies, and other institutions to perceive and act on trends and threats? Nobody really knows.

The goal of the Good Judgment Project is to answer these questions. We will systematically compare the effectiveness of different training methods (general education, probabilistic-reasoning training, divergent-thinking training) and forecasting tools (low- and high-information opinion-polls, prediction market, and process-focused tools) in accurately forecasting future events. We also will investigate how different combinations of training and forecasting work together. Finally, we will explore how to more effectively communicate forecasts in ways that avoid overwhelming audiences with technical detail or oversimplifying difficult decisions.

Over the course of each year, forecasters will have an opportunity to respond to 100 questions, each requiring a separate prediction, such as “How many countries in the Euro zone will default on bonds in 2011?” or “Will Southern Sudan become an independent country in 2011?” Researchers from the Good Judgment Project will look for the best ways to combine these individual forecasts to yield the most accurate ‘collective wisdom’ results. Participants also will receive feedback on their individual results.

All training and forecasting will be done online. Forecasters’ identities will not be made public; however, successful forecasters will have the option to publicize their own track records.

Who We Are

The Good Judgment research team is based in the University of Pennsylvania and the University of California Berkeley. The project is led by psychologists Philip Tetlock, author of the award-winning Expert Political Judgment, Barbara Mellers, an expert on judgment and decision-making, and Don Moore, an expert on overconfidence. Other team members are experts in psychology, economics, statistics, interface design, futures, and computer science.

We are one of five teams competing in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Agency). The ACE Program aims “to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts.” The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public.

-

More at their blog.

—

-

UPDATE: GMU team.

Prediction markets can harness the ‘wisdom of crowds’ to solve problems, develop products, [*] and make forecasts. These systems typically treat collective intelligence as a commodity to be mined, not a resource that can be grown and improved. That’s about to change.

–> They’ve not yet started their study, but Philip Tetlock and Justin Wolfers (et al.) have already concluded that they can improve collective intelligence. I suggest we give them the Nobel prize right away. Why wait? Sancto Subito!

[*] Did Steve Jobs use prediction markets to develop Apple’s 3 post-PC devices?

Prediction markets can harness the “wisdom of crowds” to solve problems, develop products, and make forecasts. These systems typically treat collective intelligence as a commodity to be mined, not a resource that can be grown and improved. [*] That’s about to change.

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict major trends and events around the world over the next four years.

The Good Judgment Team, based in the University of Pennsylvania and the University of California Berkeley, will be one of the five teams competing — and we’d like you to consider joining our team as a forecaster. If you’re willing to experiment with ways to improve your forecasting ability and if being part of cutting-edge scientific research appeals to you, then we want your help.

We can promise you the chance to: (1) learn about yourself (your skill in predicting and your skill in becoming more accurate over time as you learn from feedback and/or special training exercises); (2) contribute to cutting-edge scientific work on both individual-level factors that promote or inhibit accuracy and group- or team-level factors that contribute to accuracy; and (3) help us distinguish better from worse approaches to generating forecasts of importance to national security, global affairs, and economics.

[*] –> ???

–> To sign up, go to goodjudgment.info.

Despite its importance in modern life, forecasting remains (ironically) unpredictable. Who is a good forecaster? How do you make people better forecasters? Are there processes or technologies that can improve the ability of governments, companies, and other institutions to perceive and act on trends and threats? Nobody really knows.

The goal of the Good Judgment Project is to answer these questions. We will systematically compare the effectiveness of different training methods (general education, probabilistic-reasoning training, divergent-thinking training) and forecasting tools (low- and high-information opinion-polls, prediction market, and process-focused tools) in accurately forecasting future events. We also will investigate how different combinations of training and forecasting work together. Finally, we will explore how to more effectively communicate forecasts in ways that avoid overwhelming audiences with technical detail or oversimplifying difficult decisions.

Over the course of each year, forecasters will have an opportunity to respond to 100 questions, each requiring a separate prediction, such as “How many countries in the Euro zone will default on bonds in 2011?” or “Will Southern Sudan become an independent country in 2011?” Researchers from the Good Judgment Project will look for the best ways to combine these individual forecasts to yield the most accurate ‘collective wisdom’ results. Participants also will receive feedback on their individual results.

All training and forecasting will be done online. Forecasters’ identities will not be made public; however, successful forecasters will have the option to publicize their own track records.

Who We Are

The Good Judgment research team is based in the University of Pennsylvania and the University of California Berkeley. The project is led by psychologists Philip Tetlock, author of the award-winning Expert Political Judgment, Barbara Mellers, an expert on judgment and decision-making, and Don Moore, an expert on overconfidence. Other team members are experts in psychology, economics, statistics, interface design, futures, and computer science.

We are one of five teams competing in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Agency). The ACE Program aims “to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts.” The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public.

-

UPDATE: IARPA has removed the US citizenship requirement for survey participants.

More at their blog.

—

Top tier sports, national elections, and Hollywood releases are all arenas in which all the information one might analyze is already pretty much public. There are many methods for predicting these outcomes, and I wouldn’t argue that prediction markets have a huge advantage in these arenas. The markets where I expect PMs to have an advantage are where there are experts who, given an incentive, could share (or discover) information that’s not already public, and where you don’t already have an enormous crowd trying to figure out the answer. Certainly it’s fun to bet on your team or party, or to develop expertise on how the public will react to particular movies, but it’s not clear to me that we get better predictions in those areas.

This is also one of my criticisms of the Servan-Schreiber paper. While I believe there are probably markets in which the availability of serious money to be won could attract people who’d be willing to spend research in order to get a better answer, NFL sports isn’t an arena where spending thousands of dollars will help you uncover facts that aren’t already in the mainstream media.

When we talk about CEO markets, or product release dates, or market penetration numbers, we’re talking about [prediction] markets in which the information isn’t already out there, and some people will spend time and effort to ferret out relevant facts for reputation (we see this often on Foresight Exchange) or money.

Robin Hanson at Singularity Summit 2009 — How Does Society Identify Experts and When Does It Work? from Michael Anissimov on Vimeo.

More videos of Singularity Summit 2009

Singularity Summit 2009 @ Vimeo

IDEAS: Any individual expert is likely to be wrong. What happens when you put a bunch of them in the same room? Does the consensus come up with a better prediction than the individual?

TETLOCK: Usually yes. Jim Surowiecki wrote that very clever book on “The Wisdom of Crowds.” . . . Sometimes social scientists call it the miracle of aggregation. It’s an interesting fact that the average of the experts’ predictions often outperforms the individuals from whom the average was derived.

IDEAS: But the consensus can be wrong, too, as in the case of the widely shared view that Iraq had weapons of mass destruction. How does that happen?

TETLOCK: The crowd can certainly be wrong. Sometimes, the crowd can be dramatically wrong. . . . One big danger is if you’re dealing with an ideologically, intellectually homogeneous group, and they’re all talking to each other. They can really talk each other into some quite extreme positions.

-

Philip Tetlock interviewed (this year):

What makes some forecasters better than others?

The most important factor was not how much education or experience the experts had but how they thought. You know the famous line that [philosopher] Isaiah Berlin borrowed from a Greek poet, “The fox knows many things, but the hedgehog knows one big thing”? The better forecasters were like Berlin’s foxes: self-critical, eclectic thinkers who were willing to update their beliefs when faced with contrary evidence, were doubtful of grand schemes and were rather modest about their predictive ability. The less successful forecasters were like hedgehogs: They tended to have one big, beautiful idea that they loved to stretch, sometimes to the breaking point. They tended to be articulate and very persuasive as to why their idea explained everything. The media often love hedgehogs.

How do you know whether a talking head is a fox or a hedgehog?

Count how often they press the brakes on trains of thought. Foxes often qualify their arguments with “however” and “perhaps,” while hedgehogs build up momentum with “moreover” and “all the more so.” Foxes are not as entertaining as hedgehogs. But enduring a little tedium is worth it if you want realistic odds on possible futures.

So if you were looking for a money manager, you’d want a fox?

If you want good, stable long-term performance, you’re better off with the fox. If you’re up for a real roller-coaster ride, which might make you fabulously wealthy or leave you broke, go hedgehog.

But it was doomster hedgehogs like money managers Robert Rodriguez and Jeremy Grantham who first saw the crisis coming.

Hedgehogs are sometimes way, way out front. But they can also be way, way off.

-

His book: Expert Political Judgment: How Good Is It? How Can We Know?

Excellent.

Mr. Kristof, if you want to keep yourself accountable and track the success of your predictions in the long run and in real-time, why not simply participate in a prediction market such as NewsFutures?

You could suggest that particular stocks be listed in relation to particular new stories and their possible outcomes. Then, as you invest in particular outcomes, your prediction portfolio would either grow or shrink, providing us all with an objective measure of your foresight. You could feature on your blog a widget displaying in real-time the “net worth” of your various predictions.

Other advantages of this approach would include:

- Forcing a detailed specification of possible outcomes;
- Having you compete directly (bet against) the general public;
- Measuring how much your columns can influence price movements for various predictions;
- Leading by example to show other pundits how it’s done.

There are various types of prediction markets out there, so you can pick the venue where you’d feel most comfortable:

- Play-money only, like NewsFutures; [or HubDub ]
- Real betting (illegal) like Intrade;
- Charity-driven, like Bet2give.

If the idea intrigues you, please contact me at ejss@newsfutures.com and we can get you started right away!

Emile Servan-Schreiber
CEO, NewsFutures

Excellent.

Readers, do click on the link (which will bring you to the New York Times), and do click on “Recommended” under Emile’s comment —so that his pitch for the prediction markets will be more visible to all the people reading the comments there. Thanks. Appreciated.