Prediction Markets + Market Predictions = Collective Forecasting That Pays Off
IBM Smarter Cities prediction mechanisms
20 prediction lines
Only one socially generated prediction is over 50%.
UPDATE: The first version of this post used the term “prediction markets”, because that is how Spigit brands these mechanisms, but, on a closer look, each participant can’t choose the amount to bet/trade. –> FAIL. On top of that, their system is a beauty contest. –> FAIL2.
UPDATE: Jed Christiansen’s post
You can follow all the replies to this entry through the comments feed.
Jason Ruspini on OBAMACARE PREDICTION MARKET WAS BRIEFLY MANIPULATED YESTERDAY EVENING SO IT WOULD ARTIFICIALLY CLOSE AT $35. – [CHART]
Paul Hewitt on When the ObamaCare event derivative reaches 80%, I will officially declare America as a socialist country —just like France and Sweden.
Chris F. Masse on We interrupt this report to bring you some Jazz.Chris F. Masse on OBAMACARE PREDICTION MARKET WAS BRIEFLY MANIPULATED YESTERDAY EVENING SO IT WOULD ARTIFICIALLY CLOSE AT $35. – [CHART]Chris F. Masse on “InTrade should address this incident and suspend the account of whoever did this.”
Niall O'Connor on We interrupt this report to bring you some Jazz.Niall O'Connor on OBAMACARE PREDICTION MARKET WAS BRIEFLY MANIPULATED YESTERDAY EVENING SO IT WOULD ARTIFICIALLY CLOSE AT $35. – [CHART]Niall O'Connor on “InTrade should address this incident and suspend the account of whoever did this.”
Max Keiser on OBAMACARE PREDICTION MARKET WAS BRIEFLY MANIPULATED YESTERDAY EVENING SO IT WOULD ARTIFICIALLY CLOSE AT $35. – [CHART]Jason Ruspini on OBAMACARE PREDICTION MARKET WAS BRIEFLY MANIPULATED YESTERDAY EVENING SO IT WOULD ARTIFICIALLY CLOSE AT $35. – [CHART]
Interesting. Seems like a beauty contest, i.e. no grounding in real outcomes. For example, how would this outcome be measured?:
http://smartercities.ibm.spigit.com/PredictionMarket/MarketHome?marketid=23
I do not really understand why the consider these opinion polls to be “prediction markets”. Especially given the fact that they expire on September 13th 2009.
The question could be phrased “Which of the following answers will have the higher price at expiration?”
I do not call them prediction markets, for the reason mentioned by Dave above: There is no real truth to judge which contract will win.
Even if the participants can put different amount of money of the different predictions, there is no way of evaluating which prediction will be the “correct” one.
So, you are really trying to predict the results of a beauty contest, not the technology that will be proven to be the most useful.
Panos,
I read closely their rules.
http://smartercities.ibm.spigit.com/UserTab?usertab=0
Do you think it is a poll mechanism, actually?
Hi all -
Good points raised here from a knowledgeable crew. It’s true that the outcomes are not mutually exclusive, and that with a Sept 13 ending, you cannot close out the markets with a defined outcome.
A poll with advanced features is a better way to put it. For each poll, you have a finite amount to use to weight your answers. In this way, you can signal your views on the different, non-exclusive outcomes.
Given the MidasOracle prediction markets focus, I get the “FAIL” sobriquet appended by Chris. Hopefully, though, you do see some value in aggregating the perspectives of people around the world for their take on what will best help some big issues facing cities. Personally, I found the education questions valuable as a father of two.
Thanks,
Hutch
On second thought, I think you’re right, it can’t qualify as prediction markets. I will update the post.
Panos, Hutch Carpenter is going to respond here once I register him.
“A poll with advanced features is a better way to put it.”
Spigit should not brand them as “prediction markets”, then.
http://smartercities.ibm.spigit.com/UserTab?usertab=0
I gave it a fail as well: http://twitter.com/ipeirotis/status/3472223792
The fail is not due to the “MidasOracle prediction markets focus”. The #fail comes from not understanding what a prediction market is and why it is different from a poll.
Panos, I agree. The #fail comes from offering something and publicizing it another way. No good.
P.S.: I put too much weight in what Spigits says, as opposed to my critical judgment. #fail2
Sorry Chris – don’t want to do wrong by you. Will be better next time.
In my view, Spigit should:
1. Change the description of this product.
2. R&D prediction markets, and write a good prediction market software package (or license one).
Well, that’s not actually necessary. We do have good ol’ fashioned prediction markets. This particular implementation is specific to IBM. Wanna demo sometime?
Well, if Spigit could show their prediction market software to the Midas Oracle readers, that would be interesting.
Hmm…interesting Chris. What do you propose?
Now that you are registered, why don’t write a guest post about Spiit’s prediction market software? … With screen shots and examples…
Hutch,
you should call them “probability markets”, my preferred usage.
Start a revolution, don’t let this bunch of theorists get to you.
Market prices should be interpreted to reflect the probability of the parameter, not the outcome. The outcome is, by and large, irrelevant. And as for the prediction market industry, self-defeating!
Medemi, could you please explain what exactly is being traded here?
This is poll question. It is directly equivalent to the “votes” used by the UserVoice customer feedback system.
See http://feedback.mendeley.com/pages/4941-mendeley-feedback to see an example. The users of the Mendeley system can see what other people want, and place their (limited) votes accordingly. Exceptionally nice and clever: let the users decide what are the important features that should be implemented and the most pressing bugs that need to be fixed. Also let the users place different weight on different issues.
There is no need to call this a “prediction market” and UserVoice never calls their feedback system a market. They simply know what they are selling.
Panos,
if it’s a market, and it can be traded, then it’s not a poll.
Many features of prediction markets are represented in this setup.
If a trader perceives the price to be undervalued or overvalued then he will wager money assuming that the price will move in his favor. The dynamics are the same.
Consider a binary prediction market where the price is 50%. Who cares what the actual outcome will be? It’s a fifty/fifty deal and there will be NO valuable feedback from the actual outcome of this specific event.
It all boils down to the question “What makes prediction markets accurate?”
This is the question you have to answer. My conclusion is that there are more dominant/pressing issues than actually having an outcome.
I agree they shouldn’t be called prediction markets. But then I wouldn’t call prediction markets “prediction markets”. They’re probability markets.
No.
This is a poll, and the corresponding poll question is: “Which of the following answers will be the most popular”.
Actually, if you want to talk about accuracy, this is *NOT* an accurate mechanism for this purpose. In fact, a simple poll would have been better. Why? Because of the self-reinforcing nature of these contracts, the winner will get even more votes. So, the contract that gets accidentally a few votes initially, has disproportionate many chances of being the winner, even if nobody believed in this answer. The result will NOT be the representative opinion of the users, or the weighted representation. It is a fad-reinforcing mechanism.
If you want to see a study, read the study of Duncan Watts, published in Science: http://www.sciencemag.org/cgi/content/abstract/311/5762/854
They show how this social influence actually distorts the results more, compared to the case of independent choices.
And the “truth grounding” of prediction markets serves to avoid the self-reinforcement described above.
If a market, grounded on a real outcome, says A=90% and B=10% one day before expiration, and I believe that B is the real winner, then I will bet on B.
In this poll-like question, it makes no sense whatsoever to bet on B. The A will be the winner just due to the fact that the winner will be determined by which contract is leading at the expiration (in this case it will be A), not by which technology is going to be the true winner.
So, as a player I have no incentives to reveal my true beliefs. Instead I have all the incentives to follow the crowd in a bubble.
Remember: The crowd gives good decisions when each player thinks for itself and has the incentives to do so.
>> This is a poll, and the corresponding poll question is: “Which of the following answers will be the most popular”.
The only question I read was to the effect of – “Which measure will be most effective”
That’s an ok question to me.
>> Actually, if you want to talk about accuracy, this is *NOT* an accurate mechanism for this purpose. In fact, a simple poll would have been better. Why? Because of the self-reinforcing nature of these contracts, the winner will get even more votes.
Panos… it’s about making money, providing people with the proper incentive. People will not bet against the size of their wallets. Well, gamblers do some (most) of the time, and even then betfair markets are very accurate.
>> The result will NOT be the representative opinion of the users, or the weighted representation.
So what, prediction markets represent a democracy all of a sudden? You don’t have a clue what you’re talking about, IMO.
“The only question I read was to the effect of – “Which measure will be most effective”
That’s an ok question to me.”
And how are you going to verify the correctness of these predictions? By measuring the weighted popularity of the answers? Good luck with that.
“You don’t have a clue what you’re talking about, IMO.”
I do not argue against prediction markets. I argue against the “Smarter Cities” mechanism, which is NOT a market. I consider the SmarterCities mechanism a very flawed poll, which suffers from the self-reinforcing problem that I mentioned above.
Do not compare BetFair (which has contracts with verifiable outcomes) with SmarterCities (which will consider the “winner” to be the contract that has the highest price at expiration).
Let me rephrase my comment above, to avoid any confusion that may arise from the use of pronouns.
“Actually, if you want to talk about accuracy, this is *NOT* an accurate mechanism for this purpose. In fact, a simple poll would have been better. Why? Because of the self-reinforcing nature of these contracts, the winner will get even more votes.”
[should become]===>
“Actually, if you want to talk about accuracy, the **SmaterCities approach** is *NOT* an accurate mechanism for this purpose. In fact, a simple poll would have been better. Why? Because of the self-reinforcing nature of the SmarterCities contracts, the leading contract will get even more votes.”
>> And how are you going to verify the correctness of these predictions? By measuring the weighted popularity of the answers? Good luck with that.
I can’t, but then you can’t either. You think you can, but you can’t.
I’ve analyzed betfair markets accuracy by collecting massive amounts of data. I have the programming skills, and a decent understanding of statistics.
You can’t, there’s no way with the available data of real money prediction markets out there.
My point is, accuracy is not dependent on whether we have an actual result or not. It is a fundamental understanding of the dynamics of prediction markets that matters most (in our quest for accuracy). He who assumes he can measure accuracy with a limited data set has a poor understanding of statistics.
Next time that you are going to lecture someone and try to impress them with your qualifications, take a moment and see what are *their* qualifications.
“I’ve analyzed betfair markets accuracy by collecting massive amounts of data. I have the programming skills, and a decent understanding of statistics.”
Sorry, my PhD in Computer Science and my professor position in a top-10 Business School apparently are not enough. Your programming skills rule. I also have no idea about statistics.I know. I was teaching the PhD course on statistical machine learning at NYU but clearly you know better.
“You can’t, there’s no way with the available data of real money prediction markets out there.”
I have all the data from Intrade starting in 2005, crawling the data every few hours from their website, and lately using their API. I even built a public website lately for collecting all the data from Intrade and making them available for everyone to use: http://intrade-archive.appspot.com/ I also collect the data from BetFair. I had to launch a European instance of EC2 to be able to, but now I can.
But still you rule. Because you have the ability to collect “massive amount of data”. Apparently, I cannot. Sorry.
But again this is not the point, BECAUSE: ***I do not say anything against prediction markets***. Do not try to portray me as being against real prediction market, or even against play money prediction markets. These markets rule and they are accurate. Nobody argues against that. As you may (not?) know, all these markets have one thing in common: They are based on events that can be verified at expiration. The “Which measure will be most effective” CANNOT be verified. So, SmarterCities will close the market and declare as winner the contract that trades at the highest price at expiration.
I argue against calling this mechanism used in Smarter Cities a prediction market. It is NOT a prediction market. It will NOT have the predictive power of any prediction market. Why? See above.
>> Next time that you are going to lecture someone and try to impress them with your qualifications, take a moment and see what are *their* qualifications.
Don’t presume everyone is like you. I wasn’t trying to impress anyone. It’s just that sometimes, when you make a post, a little bit of background information seems appropriate.
Sorry if I’m not easily impressed. I was evaluating your posts on content.
>> I argue against calling this mechanism used in Smarter Cities a prediction market. It is NOT a prediction market.
There is no disagreement, here.
>> It will NOT have the predictive power of any prediction market.
That’s debatable. I think there could be a lot of value here. Especially when considering the self-defeating proposition of prediction markets. You know, the failure of the masses to be able to handle them (and interpret them) properly. I would think you would be able to get my point by now, if you’re so smart.
“Don’t presume everyone is like you. I wasn’t trying to impress anyone. It’s just that sometimes, when you make a post, a little bit of background information seems appropriate.”
You said that you can do the analysis (because you know programming and statistics), and that I cannot do it. So, I also gave “a little bit of background information” to demonstrate the opposite.
But let’s forget about this. No reason to continue the flame war. The focus is the SmarterCities mechanism.
“That’s debatable. I think there could be a lot of value here”
I will repeat: The SmarterCities mechanism will be just a fad-reinforcing mechanism. I gave the arguments above on why this is the case, and cited the Science article to support it. I am pretty convinced that the results of the Science article will also apply to the setting of SmarterCities.
Is there a better mechanism? I think so.
>> I will repeat: The SmarterCities mechanism will be just a fad-reinforcing mechanism.
fad-reinforcing? The SmarterCities mechanism seems more susceptible to market manipulation. But despite all the potential hazards, I suppose my point is, that if the market can be traded with substantial amounts of money then it will be very powerful almost by definition. Relative to the polls, I think the SC mechanism adds a lot of value, given the proper implementation. I don’t have all the answers at this time as to what that should entail.
Your point is markets should always have a verifiable outcome. I’m not so sure. Traders trade on expectations. Those expectations are a reflection of their perceived reality. People don’t want to bet against the “truth” or general consensus or what they perceive to be the final price! because it will cost them money. Hence, the markets will have “a tendency” to “get close to the truth”, or better, what the public perceives to be the truth. This is not some hyped housing market or oil-derivative with run-away prices due to large institutions with vested interests.
Interestingly, betfair offer a beauty contest market. It’s a once a year event and the outcome is a bit of a lottery (so it seems), so the value of having a verifiable outcome seems limited to me other than for settlement purposes of course. Yet the market is very interesting to trade as people do their upmost best to turn it into some sort of science.
I’m sorry Panos, but I do see opportunities here, given that the SC markets will provide a final price and settlement as well.
“given that the SC markets will provide a final price and settlement as well.”
Well, they say the devil is in the details: How are they going to settle a market? You need a verifiable event in order to settle the market. This is at the very core of my argument. In order to trade on expectations you need a future event that will be used for the settlement, and on which the expectations are based.
Unless I am missing something, I do not remember any market that does not have a verifiable event for settlement purposes.
SmarterCities will not have such an event. So, I suspect, when they will close the market on Sep 13th, the winner will be the contract with the higher price. Completely self-referential, and hence self-reinforcing, and hence fad-enforcing.
Just for fun, here is the only self-referential contract that I know that is *not* self-reinforcing: http://www.hubdub.com/e/Market/Will_the_no_option_be_a_greater_percentage_than_the_yes_option_at_suspend_date_3126/view
I hate to stop this interesting ping-pong between you guys – let’s try to be productive here, shall we?
of course, i fully agree – this is not a prediction market.
But – this is not the first time any of us dealing with prediction markets are faced with this challenge – the market relates to an event that is very far away, and thus it makes no sense to keep the market open until the answer is known.
So I would be very happy to hear: what solutions do you think will best fit this case?
let me offer the first twist to this:
lets say you have a simple yes/no question about an event that will occur 3 years from now.
At the expiration date, instead of determining the expiration value to be yes or no (100 or 0) – why not decide the value is the closing value? i.e., if the market said 75% for “Yes”, than that’s the actual outcome.
I know it is not ideal, but it still serves the puropose – it pays off more to bet early in the game, but it still pays off to place a bet at the last minute.
Again, same problem: It pays to go with what the crowd will believe (albeit early), not what the actual outcome will be. In other words, there is no incentive for the contrarians to bet on outcomes that are not the early favorites. In your scenario, Hillary Clinton would have won the market for democratic nominee for the 2008 election if the market had stopped in November 2007, with a winning value of 70%.
If I interpret correctly what you suggest, this will be equivalent to a “prediction market on a prediction market”, aka “options on markets”: Guess where the price of a long term market will be at set points in the future, before the expiration of the long-term market. InTrade experimented with such contracts last Fall (the X contracts). I was initially fascinated. However, we soon realized that such markets do not offer much additional information: Current price of the long-term contract and time to expiration are enough to determine the optimal price of the “X” contract. See http://bit.ly/B1W9w
The solution: Bite the bullet and have a long-term contract, based on a verifiable outcome. If you are interested in having checkpoints along the way, use conditional prediction markets (e.g., see the tax features, conditional of the result of the presidential election). Or think harder of what you are trying to measure and build a contract that has a verifiable outcome early on.
Also take a look at http://blog.mercury-rac.com/2009/08/30/starting-from-the-wrong-metaphor-prediction-markets-and-ideas/ and read the pointed Wikipedia article on Keynesian Beauty contest. You will understand why the self-referential market that you suggest is not going to be a good idea.
>> I hate to stop this interesting ping-pong between you guys – let’s try to be productive here, shall we?
Noam, you’re an idiot. You accuse me of playing ping-pong, then you steal my idea.
Pay more attention next time.
Panos, you keep missing the point, presumably because you lack the experience and/or insight.
I have nothing more to say on this issue, because you people are hopeless.
Very mature answer Medemi… congratulations.
Despite that, I’ll take my chances and try again – inviting all scholars and prediction markets expert to participate and share from your experience –
what do you find to be the best solution for such a case?
Is it like Panos says – simply wait a long time?
Medemi, I get your point, also articulated by Noam: “the market relates to an event that is very far away, and thus it makes no sense to keep the market open until the answer is known.” I do not disagree that this is a problem. I do not disagree that there is a lot of value if someone manages to solve the problem. But such “early expiration, no verifiable outcome, self-referential markets” approach is not the solution.
My point is that having self-referential markets is not working. If you assume that players will not be strategic, then yes they may work. But I want to believe that the players that participate in such markets will be strategic. See http://www.wiserthanthecrowd.com/2009/09/lolmarkets-ibm-smarter-cities.html for such an example. He simply followed the strategy that I suggested. I do not believe that he read my comments to do that; it is just so natural.
Do you disagree with the “Keynesian beauty contest” idea? http://blog.mercury-rac.com/2009/08/30/starting-from-the-wrong-metaphor-prediction-markets-and-ideas/
You even admitted yourself earlier on that the markets are prone to manipulation. Did you mean something else?
Hutch has posted this:
Spigit Prediction Markets
http://www.midasoracle.org/2009/09/10/spigit-prediction-markets/
Chris – I can do that. I’ll coordinate with you offline.