Midas Oracle .ORG » Michael Giberson

“Wisdom of crowds” in popular culture, again

Michael Giberson — Tue, 29 Sep 2009 13:56:46 +0000

“The wisdom of crowds” has apparently seeped a bit into popular culture, or at least the geekier end of it.

On the heels of British illusionist Derren Brown’s invoking of “the wisdom of crowds” as a (false) part of his explanation of how he appeared to predict winning lottery numbers, last night a character in the American TV show House invoked the wisdom of crowds as part of an explanation for how he obtained a diagnosis of his medical condition.

(The character – a highly intelligent, geeky, successful video game designer – posted his medical symptoms on the internet and offered $25,000 for a successful diagnosis. Then, mentioning “wisdom of crowd” based reasoning, concluded that the most frequent diagnosis appearing in emailed responses was likely correct. As the story turned out, the crowd-sourced diagnosis was incorrect. Instead, the correct diagnosis was submitted by series main character Greg House, working from home after quitting his job at the hospital. The “wisdom of crowds” element doesn’t make it into the official episode summary.)

Although the crowd was wrong (the better to highlight how clever our main character is when, later, he provides the correct diagnosis), at least the basic “wisdom of crowds” logic illustrated in the episode was correct. As a fan of the show, I appreciate that it doesn’t insult my intelligence by dressing up clever cons with misleading science-based patter.

Midas Oracle = About + Archives + Authors + Best + Contact + Information Technology + Links + Probabilistic Predictions + Post-Mortem Predictions + Prediction Exchanges + Prediction Software + Publish Your Post + How To Publish

Can a Keynesian beauty contest improve Pres. Obama’s suggestion box for U.S government employees?

Michael Giberson — Thu, 24 Sep 2009 13:35:10 +0000

President Barack Obama has created the “SAVE Award,” a process by which U.S. government employees can submit ideas for “how their agency can save money and perform better.” A committee of senior officials from the Office of Management and Budget at the White House will review the submissions and submit a short list to the President, and the President will pick the winner. The federal employee with the winning suggestion will get to meet the President and have the idea included in the government’s next budget.

NewsFutures suggests that the SAVE Award approach just scratches the surface of the possible benefits of crowdsourcing innovation. (Noted earlier in this post at Midas Oracle.) Here is their diagnosis and recommendation:

What’s wrong with this process?

There is no aggregation of the collective wisdom. The crowd is called on only to submit ideas, not to help evaluate them, which is the critical step in the delicate wisdom of crowd recipe. Instead, a bunch of political appointees in the OMB will decide which ideas get on the short list that the President will see. Needless to say, we would not dare propose such a simplistic process to our enterprise clients looking to harvest and select innovative ideas from their employees.

Here’s how it should be done: After everyone has proposed their ideas, ask everyone to bet (not vote) on which idea the President will ultimately select – or alternatively, on which ideas will be short-listed by the OMB officials. The crowd’s betting will quickly and efficiently produce a ranking of the best-to-worst ideas, and the result will be less arbitrary than what you can expect form a close-knit group of bureaucrats. Now, it doesn’t mean that the OMB officials cannot have the ultimate say on what ends up on the short list that goes to the Oval Office, but their choice is at least informed by the crowd’s aggregate choice.

Let’s ask NewsFutures’s question again: “What’s wrong with this process?”

Since the crowd is being asked to bet on which idea(s) will be chosen, not which ideas would actually produce the best productivity improvement, the betting will not “quickly and efficiently produce a ranking of the best-to-worst ideas.” Instead, it will produce a ranking of which items the bettors believe are more likely to be selected by the OMB and President.

If the short list was actually determined by the betting rather than independently selected by the OMB, the process takes on elements of a Keynesian beauty contest. No longer would bettors be betting on which ideas would be best, nor which idea among all submissions is most likely to be chosen by the President. Instead, the winning bettors would be those that best predicted which idea would be chosen from among those ideas most heavily favored by other bettors.

Rather than aggregating collective intelligence to identify the best idea, the process instead becomes one in which, in Keynes’ words, “we devote our intelligences to anticipating what average opinion expects the average opinion to be.” Perhaps a prediction market or other mechanism for collecting the “wisdom of the crowd” could improve the SAVE Award process, but a Keynesian beauty contest seems the wrong way to go.

[The post was originally published in slightly different form at Knowledge Problem.]

Can prediction markets help improve economic forecasts?

Michael Giberson — Mon, 15 Jun 2009 16:40:57 +0000

At VOX, David Hendry and James Reade examine the question, “How should we make economic forecasts?” Among the ideas discussed is whether prediction markets could be used to improve economic forecasting. Interesting suggestion and seeming to be worthy of additional exploration, but the authors don’t go too deep here. Instead, they assert that “prediction markets can be viewed as a form of … model averaging,” and then drift into a discussion of forecast averaging. I’m not sure that forecast averaging is a good way to look at prediction markets.

Here is what they say:

Prediction markets can be viewed as a form of forecast pooling or model averaging, a common forecast technique (Bates and Granger 1969, Hoeting et al 1999 and Stock and Watson 2004). That is, forecasts from different models are combined to produce a single forecast. In prediction markets, each market participant makes a forecast based on his or her own forecasting model, and the market price is some function of each of these individual forecasts.

Since the “prediction” implied by a prediction market is set by the marginal transaction, it depends not at all on the distribution of earlier trades, nor on the valuations of parties priced out of the market at the current price.

For example, consider two event markets: in the first 999 contracts trade at $0.50 and the 1000th and final trade is at $0.75; in the second 999 contracts trade at $0.76 and the 1000th and final trade is at $0.75. In the typical interpretation of prediction markets, the event is “predicted” to result with a 75 percent probability in both cases. However, averaging among the different predictions doesn’t get you that result.

(Well, strictly speaking the market price is “some function” of the prices – namely, one in which all trades but the last are weighted zero and the last trade is weighted one. You can call this “averaging,” but that isn’t the most useful explanation of the function.)

I’m not arguing that forecast averaging might not be a good idea in many situations, just that averaging doesn’t seem like a good way to explain what a prediction market is doing.

What’s the big deal about the fluff on prediction markets in The Economist?

Michael Giberson — Thu, 05 Mar 2009 20:10:53 +0000

Chris Masse has posted several times about the recent piece on prediction markets in The Economist. Among his recent grand pronoucements:

“If you are a prediction market consultant, and have nothing to say about the negative piece from The Economist, then you don’t matter anymore.”

Really?

What is it that The Economist said that prediction market consultants should have something to say about?

I read article when it showed up online, and then again when the print version arrived and after Chris mentioned it at Midas Oracle, and now a third time since I wonder why he is making a big deal of it.

Fortunately, it is a light read – reading it three times has not been particularly taxing. But there is not much depth to engage: An intro paragraph, some explanation, three brief examples-two of which illustrate implementation problems, and then the article concludes with a quip that “Perhaps the best way to find out when prediction markets will finally take off is to ask your employees–using a prediction market.”

That’s it.

No searing indictment of the prediction market software vendors and consulting business, no challenge to the theoretical foundations of prediction markets, no rejection of the “wisdom of the crowds.” All the article does is observe that the tool has “yet to take off.”

So my question to Chris remains, “What is it that The Economist said that prediction market consultants should have something to say about?”

(Cross-posted from Knowledge Problem.)

Where is the post-election academic conference on prediction market performance?

Michael Giberson — Tue, 07 Oct 2008 21:09:39 +0000

So, where is the post-election party going to be?

No, I don’t mean the post-election gatherings by the candidates and their supporters, I mean the post-election gathering of academics and other researchers to assess what has been learned about the performance of prediction markets over the 2008 presidential campaign cycle.

I’ve checked the usual suspects and found nothing.

(The Southern Economic Association meeting this year is in Washington, D.C., a few weeks after the election. It would be an excellent time and place for an early post-mortem on the performance of prediction markets addressing the 2008 U.S. presidential election. But a look at the program doesn’t reveal any related panels. (The deadline for submission of abstracts was April 1, 2008.) In January 2008, the American Economic Association meetings featured a panel put together by Eric Zitzewitz. The next American Economic Association meeting is in San Francisco in January, but from my scan of the preliminary program no prediction market sessions are planned. (You really have to plan ahead for the AEA meetings, the deadline was February 28.))

This ongoing election cycle has featured an explosion of interest in political prediction markets, periodic allegations of manipulation and related analysis (and here, and here, and here, and here, etc.), innovations in market design (and disputations thereof) and various claims of market failure. A lot has happened. What have we learned?

Where is the post-election academic conference on prediction market performance?

(And, in a related question, where is the post-conference party going to be?)

Quit mulling over the VP-choice prediction markets, today’s real story is in the election winner markets.

Michael Giberson — Fri, 29 Aug 2008 19:01:00 +0000

I would suggest that the VP selections and the performance of the VP-choice markets at InTrade and elsewhere lend some validity to Chris Masse’s views on such markets. But enough about the VP markets, already. The interesting developments are in the election-winner markets.

Since just after 2 PM Irish time, when the NEW.REP.VP.PALIN contract briefly fell into the 20s (rumors had it that Palin wasn’t on a plane to Dayton; subsequently established that the rumor was not true), the contract turned sharply up to about 98 and stayed there until the selection was made official.

During that same time period, the “Obama wins” contract has slipped down a few percentages and the “McCain wins” contract is up a few percentages. Since at most VP selections are typically expected to affect final vote totals by 2 or 3 percent, the fact that the Obama and McCain contracts (which are winner-take-all, not vote-share contracts) have moved by 2 or 3 percent themselves suggests the markets think Palin is a fairly strong choice.

(But as I write this, the Obama contract is rallying back. Live blogging the prediction markets is hazardous stuff.)

UPDATE: As of Tuesday morning, both presidential markets have slid back to their pre-Palin-announcement levels, but active trading suggests continued disagreement about the information trickling into the market. Also interesting, activity has continued on the NEW-REP-VP-PALIN contract, with the price dipping below 95 (but back to 97 as I write). Since that contract expires at the convention – i.e. in a day or two – some folks are betting Palin will be off the ticket fast.

What public interests are served by event contracts?

Michael Giberson — Wed, 11 Jun 2008 13:45:14 +0000

Comments continue to trickle in at the U.S. CFTC, in response to the agencies request for comments on its “concept paper on event contracts.” You can find the comments posted so far, all seven of them, on the CFTC’s website. Likely many more comments will be filed around the July 7 deadline, and show up online a few days later.

One of the questions that the CFTC asks is, “What public interests are served by event contracts?” In the most recent comment posted by the CFTC, economist John Blank addresses that question as follows:

Information aggregation is a costly exercise for private individuals, small businesses, large businesses, government entities, and policy makers operating without the benefit of public, transparent information. Critical information is provided by market transactions that are disseminated. So, matching trades for the purpose of information aggregation in event markets, when it is done in the public market context, is far more preferable than having this information available only to those who can pay for it; or who undertake a large enough, and constant enough, and varied enough, number of the relevant commercial transactions themselves.

Large companies with numerous, varied, high-stakes commercial transactions — that is to say, companies that should have a pretty good idea of the market value of the commodities that they deal with — nonetheless find it useful to spend a lot of money on industry price reports. They want to know not only what they pay and are paid, they want to know what other people are paying for the kinds of stuff they buy and sell. When commodities are traded on exchanges offering public price data, the cost of getting that data goes way down for everyone. The result is that companies large and small can use resources more efficiently, i.e., produce more things that people want in the least expensive manner.

As Blank notes elsewhere in his short comment, we have moved from an economy based predominantly on raw materials and produced goods to an economy based on information and services. Event contracts can aid in the discovery and evaluation of information about information, and in today’s economy that kind of information can be a most valuable commodity.

Is the promise of prediction markets in the United States best explored in limited, small stakes markets under a CFTC safe harbor declaration?

Michael Giberson — Fri, 16 May 2008 14:07:34 +0000

I haven’t turned up the latest edition of Science magazine mentioned yesterday by Chris F. Masse, but a version of the Science article “The Promise of Prediction Markets“, is available from the AEI Center for Regulatory and Market Studies (find link to article on this page).

The first thing of note is that the extensive list of authors overlaps that of the “Statement on Prediction Markets” of a year ago (but lacking Statement signers Daniel Kahneman, Charles R. Plott, and Shyam Sunder). A quick scan of the piece reveals it pursues that same controversial strategy as the earlier statement – seeking CFTC safe harbor protection for a limited set of small stakes, non-profit prediction market exchanges.

We would therefore urge the CFTC to explore other approaches to ensuring safe harbors, for example, formal rules or guidance approved by the commission.

We suggest that three types of entities be eligible for safe harbor treatment. The first would be not-for-profit research institutions, including universities, colleges and think tanks wishing to operate exchanges similar to the Iowa Electronic Markets. The second would be government agencies seeking to do research similar to that of nongovernmental research institutions. The third group would consist of private businesses and not-for-profits that are not primarily engaged in research, which would only be allowed to operate internal prediction markets with their employees or contractors.

In all cases, markets would be limited to small-stakes contracts. Although the definition of small stakes is somewhat arbitrary, we use the term to mean an exchange in which the total amount of capital deposited by any one participant may not exceed some modest sum, perhaps something like $2,000 per year.

The exchanges themselves would be not-for-profit but would be allowed to charge modest fees to recoup administrative and regulatory costs. Brokers and paid advisers would be barred, reducing the risks that contracts would be sold to inappropriate or vulnerable customers or that customers would be charged fees above the amounts needed to maintain the markets. Exchanges would be self-regulated, leaving them with the responsibility to make reasonable efforts to keep markets free from fraud and manipulation.

For its part, the CFTC should allow contracts that price any economically meaningful event. This definition could allow for contracts on political events, environmental risks, or economic indicators, such as those offered by the Iowa Electronic Markets, but would presumably not include contracts on the outcomes of sports events.

… Congress should support the CFTC’s efforts to develop prediction markets.

And so on.

Maybe this is the best strategy for gaining political protection for progress, it is a difficult judgment to make. But the Science magazine placement is high profile and will certainly help raise the issue among folks who might otherwise be unaware of the need for progress on this front.

Perhaps more comments after I’ve had a change to read the piece more carefully. No doubt by then Chris F. Masse and the hard-core PM crowd will have, in their wisdom, torn this timid suggestion to shreds.

The CFTC takes a necessary step toward sorting out its role with respect to prediction markets.

Michael Giberson — Sat, 10 May 2008 18:34:12 +0000

As noted here in multiple posts over the past few days, the U.S. Commodity Futures Trading Commission is thinking about prediction markets. Last week it released a “Concept Release on the Appropriate Regulatory Treatment of Event Contracts” and invited public comments on several questions as it attempts to sort out its proper role, if any, in overseeing such markets.

Formal comments are due July 7 at the CFTC. Informal comments are already springing forth: see, for example, reactions by David Pennock, Chris Masse, and Chris Hibbert. Likely more to come.

My preliminary reaction is to be encouraged. Growth of the industry in the United States is frustrated by a lack of clarity over the CFTC’s role. If real money event contract markets are to become serious business in the United States, the government will have to figure out where these markets fit in the legal and regulatory world. The CFTC is taking a step toward figuring these things out. If CFTC’s conclusions turn out to be too burdensome, then appeals can by made – to the Commission itself, to courts, or to Congress. But the process has to start somewhere.

Whether their next steps are in the right direction, who knows. I think that their steps are more likely to be in the right direction if they get high quality comments filed by prediction market proponents.

(On Knowledge Problem I’ve put up a much longer post which explains the CFTC concept paper in more detail and does a little bit of sorting through the questions asked. Readers here who have already looked at the concept paper will not learn much new, but you are still welcome to take a look.)

Who would you back, the market consensus or a book-writing pundit?

Michael Giberson — Tue, 22 Apr 2008 11:38:56 +0000

Tyler Cowen picks the market consensus over book-writing pundits:

Either the current market estimate of inflation is the best estimate available, or you know that it is wrong and you will be a very rich man. I find the former scenario more plausible.

Cowen is commenting on the Kevin Phillips book, Bad Money, recently out.

Of course book authors may be wary of going directly into the financial markets to wager their hard earned cash, which is why I have advocated prediction markets for pundits in which authors would have a chance to back their book-selling punditry with real money.

In Separating cheap talk from truly held beliefs, I wrote of political pundits with books:

Maybe they believe what they write, and would be willing to subsidize a prediction market out of their book royalties to demonstrate the strength of their convictions. Or how about the books from the current crop of U.S. presidential candidates—I wonder if these books contain any claims that are specific and substantive enough to be either true or false.

If such punditry-based prediction markets were common, mistaken-but-honest demagogues (those pundits who actually believe what they write, and are willing to stand behind it) would end up subsidizing more thoughtful analysts participating in the markets; correct honest demagogues would end up taking home larger financial rewards; and dishonest demagogues would dissemble, seek to avoid being pinned down on specific claims, and when pressed for actionable claims they would run and hide.

[Cross posted from Knowledge Problem]