Prediction Markets and the Wisdom of Crowds

1
Prediction Markets and the Wisdom of Crowds

• David Pennock
• Yahoo! Research NYC

2
Outline

• Prediction Markets Survey
• What is a prediction market?
• Examples
• Some research findings
• The Wisdom of CrowdsA Story

3
Bet Credible Opinion
Hillary Clinton will win the election
I bet 100 Hillary will win at 1 to 2 odds

• Which is more believable?More Informative?
• Betting intermediaries
• Las Vegas, Wall Street, Betfair, Intrade,...
• Prices stable consensus of a large number of
  quantitative, credible opinions
• Excellent empirical track record

4
Example March Madness
5
More Socially Redeemable Example
http//intrade.com
Screen capture 2007/05/18
6
A Prediction Market

• Take a random variable, e.g.
• Turn it into a financial instrument payoff
  realized value of variable

Bird Flu Outbreak US 2007?(Y/N)
I am entitled to
Bird FluUS 07
Bird FluUS 07
1 if
0 if
7
Why?

• Get information
• price ? probability of uncertain event(in
  theory, in the lab, in the field, ...more later)
• Is there some future event youd like to
  forecast?A prediction market can probably help

8
A Prediction Market

• Take a random variable, e.g.
• Turn it into a financial instrument payoff
  realized value of variable

US08Pres Dem?
2008 CAEarthquake?
9
Aside Terminology

• Key aspect payout is uncertain
• Called variously asset, security, contingent
  claim, derivative (future, option), stock,
  prediction market, information market, gamble,
  bet, wager, lottery
• Historically mixed reputation
• Esp. gambling aspect
• A time when options were frowned upon
• But when regulated serve important social roles...

10
Getting Information

• Non-market approach ask an expert
• How much would you pay for this?
• A 5/36 ? 0.1389
• caveat expert is knowledgeable
• caveat expert is truthful
• caveat expert is risk neutral, or RN for 1
• caveat expert has no significant outside stakes

11
Getting Information

• Non-market approach pay an expert
• Ask the expert for his report r of the
  probability P( )
• Offer to pay the expert
• 100 log r if
• 100 log (1-r) if
• It so happens that the expert maximizes expected
  profit by reporting r truthfully
• caveat expert is knowledgeable
• caveat expert is truthful
• caveat expert is risk neutral, or RN
• caveat expert has no significant outside stakes

logarithmic scoring rule, a proper scoring
rule
12
Getting Information

• Market approach ask the publicexperts
  non-experts alikeby opening a market
• Let any person i submit a bid order an offer to
  buy qi units at price pi
• Let any person j submit an ask order an offer
  to sell qj units at price pj(if you sell 1 unit,
  you agree to pay 1 if )
• Match up agreeable trades (many poss. mechs...)

13
Non-Market Alternatives vs. Markets

• Opinion poll
• Sampling
• No incentive to be truthful
• Equally weighted information
• Hard to be real-time
• Ask Experts
• Identifying experts can be hard
• Incentives
• Combining opinions can be difficult

• Prediction Markets
• Self-selection
• Monetary incentive and more
• Money-weighted information
• Real-time
• Self-organizing

14
Non-Market Alternatives vs. Markets

• Machine learning/Statistics
• Historical data
• Past and future are related
• Hard to incorporate recent new information

• Prediction Markets
• No need for data
• No assumption on past and future
• Immediately incorporate new information

15
Function of Markets 2 Risk Management

• If is bad for me,
• I buy a bunch of
• If my house is struck by lightening, I am
  compensated.

16
Risk Management Examples

• Insurance
• I buy car insurance to hedge the risk of accident
  
• Futures
• Farmers sell soybean futures to hedge the risk of
  price drop
• Options
• Investors buy options to hedge the risk of stock
  price changes

17
Financial Markets vs. Prediction Markets
Financial Markets Prediction Markets
Primary Social welfare (trade)Hedging risk Information aggregation
Secondary Information aggregation Social welfare (trade)Hedging risk
18
Giving/Getting Information

• What you can say/learn chance that
• Hillary wins
• GOP wins Texas
• YHOO stock gt 30
• Duke wins tourney
• Oil prices fall
• Heat index rises
• Hurricane hits Florida
• Rains at place/time

• Where
• IEM, Intrade.com
• Intrade.com
• Stock options market
• Las Vegas, Betfair
• Futures market
• Weather derivatives
• Insurance company
• Weatherbill.com

19
http//tradesports.com
http//intrade.com
Screen capture 2007/05/18
20
Intrade Election Coverage
21
http//www.biz.uiowa.edu/iem
22
http//www.wsex.com/
http//www.hedgestreet.com/
Screen capture 2007/05/18
Screen capture 2007/05/18
23
Play moneyReal predictions
http//www.hsx.com/
24
http//us.newsfutures.com/
http//www.ideosphere.com
Cancercuredby 2010
Machine Gochampionby 2020
25
Yahoo!/OReilly Tech Buzz Game
http//buzz.research.yahoo.com/
26
An Incomplete List of Prediction Markets

• Real Money
• Iowa Electronic Markets (IEM), http//www.biz.uiow
  a.edu/iem/
• TradeSports, http//www.tradesports.com
• InTrade, http//www.intrade.com
• Betfair, http//www.betfair.com/
• Gambling markets? sports betting, horse racetrack
  
• Play Money
• Hollywood Stock Exchange (HXS),
  http//www.hsx.com/
• NewsFutures, http//www.newsfutures.com
• Yahoo!/OREILLY Tech Buzz Game,
  http//buzz.research.yahoo.com
• World Sports Exchange (WSE), http//www.wsex.com/
  
• Foresight Exchange, http//www.ideosphere.com/
• Inkling Markets http//inklingmarkets.com/
• Internal Prediction Markets
• HP, Google, Microsoft, Eli-Lilly, Corning

27
More Prediction Market Games

• BizPredict.com
• CasualObserver.net
• FTPredict.com
• InklingMarkets.com
• ProTrade.com
• StorageMarkets.com
• TheSimExchange.com
• TheWSX.com
• Alexadex, Celebdaq, Cenimar, BetBubble,
  Betocracy, CrowdIQ, MediaMammon,Owise,
  PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds
• http//www.chrisfmasse.com/3/3/markets/Play-Money
  _Prediction_Markets

28
Catalysts

• Markets have long history of predictive accuracy
  why catching on now as tool?
• No press is bad press Policy Analysis Market
  (terror futures)
• Surowiecki's Wisdom of Crowds
• Companies
• Google, Microsoft, Yahoo! CrowdIQ, HSX,
  InklingMarkets, NewsFutures
• Press BusinessWeek, CBS News, Economist,
  NYTimes, Time, WSJ, ...http//us.newsfutures.com/
  home/articles.html

29
Does it work?

• Yes, evidence from real markets, laboratory
  experiments, and theory
• Racetrack odds beat track experts Figlewski
  1979
• Orange Juice futures improve weather forecast
  Roll 1984
• I.E.M. beat political polls 451/596 Forsythe
  1992, 1999Oliven 1995Rietz 1998Berg
  2001Pennock 2002
• HP market beat sales forecast 6/8 Plott 2000
• Sports betting markets provide accurate forecasts
  of game outcomes Gandar 1998Thaler
  1988Debnath EC03Schmidt 2002
• Market games work Servan-Schreiber 2004Pennock
  2001
• Laboratory experiments confirm information
  aggregationPlott 198219881997Forsythe
  1990Chen, EC01
• Theory rational expectations Grossman
  1981Lucas 1972
• More later

30
Does it work?Yes...

• Evidence from real markets, laboratory
  experiments, and theory indicate that markets are
  good at gathering information from many sources
  and combining it appropriately e.g.
• Markets like the Iowa Electronic Market predict
  election outcomes better than pollsForsythe
  1992, 1999Oliven 1995Rietz 1998Berg
  2001Pennock 2002
• Futures and options markets rapidly incorporate
  information, providing accurate forecasts of
  their underlying commodities/securitiesSherrick
  1996Jackwerth 1996Figlewski 1979Roll
  1984Hayek 1945
• Sports betting markets provide accurate forecasts
  of game outcomes Gandar 1998Thaler
  1988Debnath EC03Schmidt 2002

31
Does it work?Yes...

• E.g. (contd)
• Laboratory experiments confirm information
  aggregationPlott 198219881997Forsythe
  1990Chen, EC-2001
• And field tests Plott 2002
• Theoretical underpinnings rational
  expectationsGrossman 1981Lucas 1972
• Procedural explanation agents learn from
  pricesHanson 1998Mckelvey 1986Mckelvey
  1990Nielsen 1990
• Proposals to use information markets to help
  science Hanson 1995, policymakers, decision
  makers Hanson 1999, government Hanson 2002,
  military DARPA FutureMAP, PAM
• Even market games work! Servan-Schreiber
  2004Pennock 2001

32
Example IEM 1992
Source Berg, DARPA Workshop, 2002
33
Example IEM
Source Berg, DARPA Workshop, 2002
34
Example IEM
Source Berg, DARPA Workshop, 2002
35
Example IEM
Source Berg, DARPA Workshop, 2002
36
Example IEM
Source Berg, DARPA Workshop, 2002
37
Speed TradeSports
Source Wolfers 2004
Contract Pays 100 if Cubs win game 6 (NLCS)
Price of contract (Probability that Cubs win)
Fan reaches over and spoils Alous catch. Still
1 out.
Cubs are winning 3-0 top of the 8th1 out.
The Marlins proceed to hit 8 runs in the 8th
inning
Time (in Ireland)
38
Does money matter? Play vs real, head to head

• Experiment
• 2003 NFL Season
• ProbabilitySports.com Online football forecasting
  competition
• Contestants assess probabilities for each game
• Quadratic scoring rule
• 2,000 experts, plus
• NewsFutures (play )
• Tradesports (real )
• Used last trade prices

• Results
• Play money and real money performed similarly
• 6th and 8th respectively
• Markets beat most of the 2,000 contestants
• Average of experts came 39th (caveat)

Electronic Markets, Emile Servan-Schreiber,
Justin Wolfers, David Pennock and Brian Galebach
39
(No Transcript)
40
Does money matter? Play vs real, head to head
StatisticallyTS NFNF gtgt Avg TS gt Avg
41
Real marketsvs. market games
IEM
HSX
averagelog score
arbitrageclosure
42
Real marketsvs. market games
HSX
FX, F1P6
forecast source avg log score F1P6 linear
scoring -1.84 F1P6 F1-style scoring -1.82 betting
odds -1.86 F1P6 flat scoring -2.03 F1P6 winner
scoring -2.32
expectedvalueforecasts489 movies
43
A Wisdom of Crowds Story
Story
1/7
Survey
Research
Opinion

• ProbabilitySports.com
• Thousands of probability judgments for sporting
  events
• Alice Jets 67 chance to beat Patriots
• Bob Jets 48 chance to beat Patriots
• Carol, Don, Ellen, Frank, ...
• Reward Quadratic scoring ruleBest probability
  judgments maximize expected score

44
Individuals

• Most individuals are poor predictors
• 2005 NFL Season
• Best 3747 points
• Average -944 Median -275
• 1,298 out of 2,231 scored below zero(takes
  work!)

45
Individuals

• Poorly calibrated (too extreme)
• Teams given lt 20 chance actually won 30 of the
  time
• Teams given gt 80 chance actually won 60 of the
  time

46
The Crowd

• Create a crowd predictor by simply averaging
  everyones probabilities
• Crowd 1/n(Alice Bob Carol ... )
• 2005 Crowd scored 3371 points(7th out of 2231)
  !
• Wisdom of fools Create a predictor by averaging
  everyone who scored below zero
• 2717 points (62nd place) !
• (the best fool finished in 934th place)

47
The Crowd How Big?
Morehttp//blog.oddhead.com/2007/01/04/the-wisdo
m-of-the-probabilitysports-crowd/http//www.overc
omingbias.com/2007/02/how_and_when_to.html
48
Can We Do Better? ML/Stats
Dani et al. UAI 2006

• Maybe Not
• CS experts algorithms
• Other expert weights
• Calibrated experts
• Other averaging fns (geo mean, RMS, power means,
  mean of odds, ...)
• Machine learning (NB, SVM, LR, DT, ...)
• Maybe So
• Bayesian modeling EM
• Nearest neighbor (multi-year)

49
Can we do better? Markets