President Barack Obama has created the “SAVE Award,” a process by which U.S. government employees can submit ideas for “how their agency can save money and perform better.” A committee of senior officials from the Office of Management and Budget at the White House will review the submissions and submit a short list to the President, and the President will pick the winner. The federal employee with the winning suggestion will get to meet the President and have the idea included in the government’s next budget.
NewsFutures suggests that the SAVE Award approach just scratches the surface of the possible benefits of crowdsourcing innovation. (Noted earlier in this post at Midas Oracle.) Here is their diagnosis and recommendation:
What’s wrong with this process?
There is no aggregation of the collective wisdom. The crowd is called on only to submit ideas, not to help evaluate them, which is the critical step in the delicate wisdom of crowd recipe. Instead, a bunch of political appointees in the OMB will decide which ideas get on the short list that the President will see. Needless to say, we would not dare propose such a simplistic process to our enterprise clients looking to harvest and select innovative ideas from their employees.
Here’s how it should be done: After everyone has proposed their ideas, ask everyone to bet (not vote) on which idea the President will ultimately select – or alternatively, on which ideas will be short-listed by the OMB officials. The crowd’s betting will quickly and efficiently produce a ranking of the best-to-worst ideas, and the result will be less arbitrary than what you can expect form a close-knit group of bureaucrats. Now, it doesn’t mean that the OMB officials cannot have the ultimate say on what ends up on the short list that goes to the Oval Office, but their choice is at least informed by the crowd’s aggregate choice.
Let’s ask NewsFutures’s question again: “What’s wrong with this process?”
Since the crowd is being asked to bet on which idea(s) will be chosen, not which ideas would actually produce the best productivity improvement, the betting will not “quickly and efficiently produce a ranking of the best-to-worst ideas.” Instead, it will produce a ranking of which items the bettors believe are more likely to be selected by the OMB and President.
If the short list was actually determined by the betting rather than independently selected by the OMB, the process takes on elements of a Keynesian beauty contest. No longer would bettors be betting on which ideas would be best, nor which idea among all submissions is most likely to be chosen by the President. Instead, the winning bettors would be those that best predicted which idea would be chosen from among those ideas most heavily favored by other bettors.
Rather than aggregating collective intelligence to identify the best idea, the process instead becomes one in which, in Keynes’ words, “we devote our intelligences to anticipating what average opinion expects the average opinion to be.” Perhaps a prediction market or other mechanism for collecting the “wisdom of the crowd” could improve the SAVE Award process, but a Keynesian beauty contest seems the wrong way to go.
[The post was originally published in slightly different form at Knowledge Problem.]
Read again, Michael, we do not propose that the short list be determined by the betting, so your “Keynesian beauty contest” objection is irrelevant in this case.
In our design, the bettors would be betting on what the OMB advisers would decide to put on the short list, presumably according to the particular intrinsic and perceived merits of each idea in regard to pre-specified requirements (listed on the SAVE Award home page). A bettor’s task in this case is to divine the ultimate selection of the advisers, not the preference of other bettors. The ranking produced by the betting – quickly and efficiently – just serves to inform the decision makers and make their selection task easier.
In the private sector, this design works beautifully. There’s no reason why it would not work for the Administration as well.
PS: NewsFutures is spelled with an “s” as the end.
There is no contradiction. People are betting on which ideas the OMB officials will put on the short list. Therefore the betting can *inform* those officials and help them in their selection task by suggesting a ranking of ideas, but the bettors don’t *decide* what goes on the short list. This ultimate decision remains with the OMB officials.
“PS: NewsFutures is spelled with an “s” as the end.”
I have corrected the misspelling.
Thanks Chris. I caught my typo in the Knowledge Problem post, but somehow missed it when posting here.
Emile, I can’t manage to reconcile your 2 statements:
- “we do not propose that the short list be determined by the betting”
- “After everyone has proposed their ideas, ask everyone to bet (not vote) on which idea the President will ultimately select – or alternatively, on which ideas will be short-listed by the OMB officials.”
There is something I don’t get, as these 2 statements seem contradictory. Am I missing something? (probably)
Emile, I agree with Michael on this. Betting on ideas that the OMB would ultimately select does not necessarily mean that traders will select ideas that will produce the best productivity improvement. Big, game-changing ideas may be discredited because traders feel that the OMB may rule them out for being overly ambitious or to disruptive. What would make sense, in my opinion, is to have the crowd generate ideas and choose the short list that the OMB can refine and submit for selection by the President. Ideally, two crowds would participate: one to submit ideas and a second to choose the short list.
In this particular case, a voting or token mechanism may work. The crowd has skin in the game because ultimately, we all want a more productive, efficient government. If the crowd knows that they can determine the short list, they should have ample motivation for providing honest recommendations. Sure, there will be some degree of “noise”, but the most effective ideas (in the crowd’s opinion) should rise to the top.
My argument with Michael is that he mistakenly called our design a “Keynesian beauty contest” whereas it’s anything but. At a minimum he should recognize his mistake and retract his dismissive prose.
Ideally, as you argue, the betting would be on the ultimate outcomes of the ideas if implemented. However, that’s not a realistic design because measuring the outcomes would be too long term to motivate the participants.
It’s not realistic because Obama will only choose one idea to move forward with. So betting on the outcomes of these ideas will not create a resolvable market.
But my suggestions is not for a prediction market. The crowd(s) should generate and select the ideas for consideration, allowing the OMB to refine their selections and ultimately allowing Obama to choose a winner. How to aggregate the crowd’s intelligence is what we’re debating and my thought here is that a poll or voting mechanism may be the best fit.
Emile, I agree that your proposal was not exactly a Keynesian beauty contest. I tried to be careful in my explanation by writing “If the short list was actually determined by the betting rather than independently…”, then the process turns into a Keynesian beauty contest.
As you presented the idea, you suggest betting on what the OMB will short list, or what the President will pick. Given the inherently political nature of OMB and Presidential decisionmaking, these picks will not necessarily be the best idea on the merits but instead the best idea as constrained by the politics. I suppose the political constraint is inherent in governmental decisionmaking – and certainly not caused by your proposed mechanism – but my first point was only that the best betting would similarly not aim for the merits but rather aim for the political choice.
However, if the OMB’s choice of short list is, as you put it, “informed by the crowd’s aggregate choice,” then a bit of the beauty contest creeps back in because the payoff becomes linked to which ideas are most favored in the betting. My second point is a conjecture that only if the OMB’s selection process is independent of betting results does the process fully escape the beauty contest problem.
You of course have practical experience using related approaches, and my commentary is more theoretical. In practice, there may be factors which prevent the mechanism from devolving in the beauty contest fashion.
I just noticed that Jed Christiansen also invoked a “Keynesian beauty contest” critique in response to Chris Masse’s post on the “IBM Smarter Cities” mechanism. That post yielded 39 comments (so far), mostly on the appropriateness of the “prediction market” label for the information aggregation tool employed. Jed points out a problem with using a prediction market to forecast the “best ideas” rather the the “best outcomes,” and suggests that “best idea” type markets should be tied to long-term project developments and results.
Jed’s post: Starting from the wrong metaphor – Prediction Markets and Ideas
Chris’s post: Are IBM Smarter Cities prediction markets too smart for people?
Jason Ruspini’s comment, here:
http://www.midasoracle.org/2009/09/24/deep-throats-take-of-emiles-take-of-the-save-program/#comment-25671
The IBM SmarterCities was absolutely a beauty contest, as the winners were decided by total amount that had been “bet” on each outcome.
There is a grey middle ground, which is essentially down to the internal mechanisms of the organization running the market. Are they lazy and only ever look at the outcomes that get a lot of votes? Then that is very close to the beauty contest. Or do they process everything separately and come to their own conclusion, no matter what the market says? Then that would be a classical prediction market.
The devil is in these details. And unfortunately, the details are rarely discussed, making threads like these a bit more opaque.
“grey middle ground”
“close to the beauty contest”
Jed’s comment resembles Jason’s comment:
http://www.midasoracle.org/2009/09/24/deep-throats-take-of-emiles-take-of-the-save-program/#comment-25671
To be honest, I hadn’t read Jason’s comment before I wrote mine. Now that I have, I think we’re generally on the same page. Many of these mechanisms are in a place that is neither a strict beauty contest nor a strict prediction market. Depending on how the decision-makers are influenced by the market outcomes determines which way (and to what degree) the mechanism falls.
I can understand Emile’s irritation of the “Deep Throat” comment. A “Deep Throat” may be useful where there’s inside information that would otherwise not be known (aka back-room CFTC issues), and would be hazardous for that individual to reveal. But when it comes to simple commentary, it’s cowardly to stand behind a pseudonym.
“I think we’re generally on the same page.”
Good, so I was right to publish that comment, then. [Few comments on Midas Oracle are anonymous. When they are, they are sometimes helpful in starting a discussion, as in this case.]
I try not to post or comment on blogs during the working day. In the future, I’ll wait to comment in my own name to avoid misunderstandings.
I share disdain for anonymity that provides cover for gutter-sniping, particularly when facts are stylized. Sometimes the “Tyler Durden” @ Zerohedge guy goes in that direction.
“Tyler Durden” @ Zerohedge”
Where does this guy post comments?
P.S.: You didn’t commit any crime, since you sticked with your anonymous comment. All was ethical. Emile drinks too much coffee.
“Tyler Durden” at zerohedge.com:
http://www.zerohedge.com/article/nicolas-sarkozy-joins-afl-cio-demanding-tobin-tax-all-financial-institutions
That post particularly annoys me. He says something untrue – that only large, institutional high frequency traders would be affected by tobin tax – that is very aggressive towards a swath of his readers and simultaneously suggests a lack of experience. (Not to mention the later naive-sounding comment about the politics of the situation, which I really hope was just sarcasm.) Overall, you wonder whether he’s more interested in reporting on finance or stirring up controversies to charge his self-aggrandizement capacitor.