Tag Archives: accuracy

Assessing the usefulness of enterprise prediction markets

Posted on by
Reply

No Gravatar

Do you need to have experience in running an enterprise prediction exchange in order to assess the pertinence of enterprise prediction markets?

Paul Hewitt:

Hi Jed…

As for qualifications, I have been making business decisions for almost 30 years. I am a chartered accountant and a business owner. Starting in university and continuing to this day, I have been researching information needs for corporate decision making. As Chris points out, I’m not a salesperson for any of the software developers. In fact, if I have a bias, it is to be slightly in favour of prediction markets. That said, I still haven’t seen any convincing evidence that they work as promised by ANY of the vendors.

As for whether I have ever run or administered a prediction market, the answer is no. Does that mean I am not qualified to critique the cases that have been published? Hardly. You don’t have to run a PM to know that it is flawed. Those that do, end up trying to justify minuscule “improvements” in the accuracy of predictions. They also fail to consider the consistency of the predictions. Without this, EPMs will never catch on. Sorry, but that is just plain common sense.

The pilot cases that have been reported are pretty poor examples of prediction market successes. In almost every case, the participants were (at least mostly) the same ones that were involved with internal forecasting. The HP markets, yes, the Holy Grail of all prediction markets, merely showed that prediction markets are good at aggregating the information already aggregated by the company forecasters! They showed that PMs are only slightly better than other traditional methods – and mainly because of the bias reduction. Being slightly better is not good enough in the corporate world.

I think I bring a healthy skepticism to the assessment of prediction markets. I truly want to believe, but I need to be convinced. I am no evangelist, and there is no place for that in scientific research. Rather than condemn me for not administering a PM, why not address the real issues that arise from my analyses?

Paul Hewitt&#8217-s blog

Previously: The truth about CrowdClarity’s extraordinary predictive power (which impresses Jed Christiansen so much)

The Accuracy Of Prediction Markets

Posted on by
24

– A Lesson in Prediction Markets from the Game of Craps &#8211- by Paul Hewitt

– Why Public Prediction Markets Fail &#8211- by Paul Hewitt

Both articles are required reading for Jed Christiansen and Panos Ipeirotis (alias &#8220-Prof Panos&#8221-). :-D

Flawed New Hampshire polls = Non-accurate New Hampshire prediction markets

Posted on by
Reply

No Gravatar

The most comprehensive analysis ever conducted of presidential primary polls:

&#8220-a handful of methodological missteps and miscalculations combined to undermine the accuracy of predictions about presidential primary winners in New Hampshire and three other states.&#8221-

Via Mister the Great Research Scientist David Pennock &#8211-who is an indispensable element of the field of prediction markets.

As I blogged many times, prediction markets react to polls&#8230- See the addendum below&#8230- – [UPDATE: See also Jed’s comment.] – Prediction markets should not be hyped as crystal balls, but simply as an objective and continuous way to aggregate expectations. So, if you think of it, their social utility is much smaller than what the advocates of the &#8220-idea futures&#8221-, &#8220-wisdom of crowds&#8221- or &#8220-collective intelligence&#8221- concepts told us. Much, much, much, much smaller&#8230- They all make the mistake to put accuracy forward. (By the way, somewhat related to that issue, please go reading the dialog between Robin Hanson and Emile Servan-Schreiber.)

–

Addendum

–

California Institute of Technology economist Charles Plott:

What you&#8217-re doing is collecting bits and pieces of information and aggregating it so we can watch it and understand what people know. People picked this up and called it the &#8220-wisdom of crowds&#8221- and other things, but a lot of that is just hype.

–

New Hampshire – The Democrats

–

The Hillary Clinton event derivative was expired to 100.

Dem NH Clinton

Dem NH Obama

Dem NH Edwards

–

New Hampshire – The Republicans

–

The John McCain event derivative was expired to 100.

Rep NH McCain

Rep NH Romney

Rep NH Huckabee

Rep NH Giuliani

–

No TweetBacks yet. (Be the first to Tweet this post)

Some enterprise prediction markets work very well… -some others are just a waste of time.

Posted on by
2

No Gravatar

Jed Christiansen:

[…] When it comes to the first point, forecasting something that the company already forecasts, prediction markets may or may not be an excellent solution. I’ve seen one set of markets that absolutely blew away the accuracy of current forecasts, and I’ve seen other markets that were consistent with current forecasts with little or no accuracy edge. […]

Care to say more about what is the determinant of an EPM success?

Enterprise prediction markets: Usability innovation is the answer.

Posted on by
4

No Gravatar

This past week, The Economist wrote on the yet-unfulfilled promise of prediction markets. At CrowdCast (ex-Xpree), we believe prediction markets are not yet mainstream because the current solutions are built on mechanisms designed for the stock market, not for the enterprise.

The stock trading metaphor works for a large, liquid stock market, but is unsuitable for enterprise forecasting. The concept of shorting and covered calls is far from intuitive for your average employee, and the stock mechanism makes it hard to ask the simplest of questions relevant for corporate forecasters. For example, buying or selling a collection of virtual stocks representing probabilities of sales falling in particular ranges is an incredibly obtuse way of asking for a single sales forecast. Finally, the stock mechanism relies on copious liquidity to ensure meaningful metrics, which is often not available with the limited crowds available in the enterprise.

However, innovation moves on and we question the assumption that prediction markets have to rely on the stock market analogy. At CrowdCast, we have been working on a new mechanism, that takes into account participant behavior and aptitude as much as market efficiency. The product we are launching in April will deliver easy, engaging, and expressive information exchanges, without the limitations of traditional notions of stock markets.

When you get the questions, incentives, and mechanism right, a prediction market can be an incredibly powerful management tool. Employees share insights anonymously and are measured and rewarded for their intelligence. Widely deployed, this has the potential to fundamentally change the nature of the organizational contract, moving from information flow based on hierarchy and silos, to enterprise-wide direct communication.

A whole new take on prediction markets- available from CrowdCast in April 2009.

Mat Fogarty

CrowdCast CEO

Cross-posted from the Xpree blog

Previously: Are collective intelligence solutions being oversold?

Inkling Markets CEO Adam Siegel speaks out on the current state of enterprise prediction markets.

Posted on by
1

No Gravatar

Adam Siegel:

Niall,

You are right to question this stuff. There is a lot of bullshit out there and frankly I cringe when I see articles or statements about the “accuracy” of markets because it hurts everyone in the long run. It’s why we’ve written 3 or 4 times on our blog about what “accuracy” in a marketplace really means – that you can’t just look at 5 or 10 markets and say “we nailed it.” Unfortunately it’s something we have to deal with because the application is called a “prediction” market after all so it’s the first question that naturally comes to mind. That said I wish Chris wouldn’t make blanket statements [*] about “the vendors, e.g. Inkling Markets” getting the use cases all wrong in enterprise prediction markets. Because frankly, for quite some time “accuracy” has been a secondary argument to a number of other advantages we discuss about markets (in fact on the page on our website that describes the value proposition to companies, we don’t even list “accuracy” as a key benefit). This is also an expectation we set with our clients right from the beginning.

Anyways, I made the statement about business increasing year over year based on our own experience/numbers and as Jed mentioned, by looking at the activity of some of our competitors who have made hires, added clients, etc. I also base the statement on the types of professional services companies we are working with/hearing from and our discussions about prediction markets and what they are going to try and do in the future. I don’t think so many would be interested in adding markets to their toolset and expending resources putting together offerings if they didn’t see them as a long term, worthwhile business capability. So for those that agree with Chris that 6 years is enough time to evaluate a new business capability, I’d like to politely disagree. I could be wrong but these things haven’t been used beyond the experimental/pilot stage in companies for more than 2 or 3 years. We’re just at the cusp of understanding what benefits they’re going to bring. We’ve seen some promising trends, we’ve also seen people try to use Inkling for something and it failed miserably. This is just standard lifecycle stuff, especially for a capability that is designed, as I said in my blog post, to bring about more transparency and break down organization barriers.

[*] Adam, it was not a &#8220-statement&#8221-, it was a 2-choice hypothesis.

ADDENDUM

An uncertain future – A novel way of generating forecasts has yet to take off. – by The Economist – 2009-02-26

Panos Ipeirotis wants HubDub (and others) to set up meta-markets -as an anti-bragging medecine.

Posted on by
16

No Gravatar

Panos Ipeirotis:

Actually, I would like to see for such collective events (senate races, oscars etc), to see a “meta-market” in each exchange:

“How many events is our exchange going to guess correctly?”

In principle, it should be possible to price correctly such contracts using the reported probabilities and their dependencies. This should give some extra attention to the fact that markets are not supposed to get everything correctly to be accurate.

Panos Ipeirotis&#8217-s blog [*] and website

[*] Great name (&#8221-Behind The Enemy&#8217-s Lines&#8221-) &#8212-but where does the name come from?

HubDub takes home the Gold at the Oscars.

Posted on by
25

No Gravatar

Well the Oscars have come and gone and although most pundits are chattering about Slumdog&#8217-s amazing 8 wins and Hugh Jackman&#8217-s dulcet tones, I&#8217-m more impressed by Hubdub&#8217-s amazing success.

Out of the major races, we got EVERY SINGLE ONE RIGHT. We were also the only major prediction exchange to correctly predict the Best Actor race (Betfair, HSX, InTrade, Newsfutures and even Nate Silver all gave the gold to Mickey Rourke). Also, in five of the big 6 races, we showed higher confidence than InTrade predictors.

HubdubInTrade
Best Picture98%90%
Best Director76%90%
Best Actor63%33.5% (wrong)
Best Actress87%85%
Best Sup Actor100%95%
Best Sup Actress64%58.8%

From the complete 24 award lineup, we nailed 19, generally by impressive condifence margins. Check out all of our settled markets here.

Not only have Hubdubbers had a successful night, each of my personal Oscar predictions were correct and I added another 40 thousand Hubbucks to my coffers. Award season is now finally behind us, but American Idol is just getting started!

Crossposted from Newspundits