Prediction%20Markets%20and%20the%20Wisdom%20of%20Crowds

1
Prediction Markets and the Wisdom of Crowds

• David Pennock, Yahoo! Research
• Joint with
• Yiling Chen, Varsha Dani, Lance Fortnow, Ryan
  Fugger, Brian Galebach, Arpita Ghosh, Sharad
  Goel, Mingyu Guo, Joe Kilian, Nicolas Lambert,
  Omid Madani, Mohammad Mahdian, Eddie Nikolova,
  Daniel Reeves, Sumit Sanghai, Mike Wellman, Jenn
  Wortman

2
Bet Credible Opinion
Obama will win the 2008 US Presidential election
I bet 100 Obama will win at 1 to 2 odds

• Which is more believable?More Informative?
• Betting intermediaries
• Las Vegas, Wall Street, Betfair, Intrade,...
• Prices stable consensus of a large number of
  quantitative, credible opinions
• Excellent empirical track record

3
A Prediction Market

• Take a random variable, e.g.
• Turn it into a financial instrument payoff
  realized value of variable

Bin Laden captured in 2008?(Y/N)
I am entitled to
Bin Ladencaught 08
Bin Ladencaught 08
1 if
0 if
4
http//intrade.com
5
Outline

• The Wisdom of Crowds
• The Wisdom of Markets
• Prediction MarketsExamples Research
• Does Money Matter?
• Combinatorial Betting

Story
Survey
Research
Research
6
A WOC Story
Story
1/7
Survey
Research
Opinion

• ProbabilitySports.com
• Thousands of probability judgments for sporting
  events
• Alice Jets 67 chance to beat Patriots
• Bob Jets 48 chance to beat Patriots
• Carol, Don, Ellen, Frank, ...
• Reward Quadratic scoring ruleBest probability
  judgments maximize expected score

7
Individuals

• Most individuals are poor predictors
• 2005 NFL Season
• Best 3747 points
• Average -944 Median -275
• 1,298 out of 2,231 scored below zero(takes
  work!)

8
Individuals

• Poorly calibrated (too extreme)
• Teams given lt 20 chance actually won 30 of the
  time
• Teams given gt 80 chance actually won 60 of the
  time

9
The Crowd

• Create a crowd predictor by simply averaging
  everyones probabilities
• Crowd 1/n(Alice Bob Carol ... )
• 2005 Crowd scored 3371 points(7th out of 2231)
  !
• Wisdom of fools Create a predictor by averaging
  everyone who scored below zero
• 2717 points (62nd place) !
• (the best fool finished in 934th place)

10
The Crowd How Big?
Morehttp//blog.oddhead.com/2007/01/04/the-wisdo
m-of-the-probabilitysports-crowd/http//www.overc
omingbias.com/2007/02/how_and_when_to.html
11
Can We Do Better? ML/Stats
Dani et al. UAI 2006

• Maybe Not
• CS experts algorithms
• Other expert weights
• Calibrated experts
• Other averaging fns (geo mean, RMS, power means,
  mean of odds, ...)
• Machine learning (NB, SVM, LR, DT, ...)
• Maybe So
• Bayesian modeling EM
• Nearest neighbor (multi-year)

12
Can we do better? Markets
13
Prediction MarketsExamples Research
14
The Wisdom of CrowdsBacked in dollars

• What you can say/learn chance that
• Obama wins
• GOP wins Texas
• YHOO stock gt 30
• Duke wins tourney
• Oil prices fall
• Heat index rises
• Hurricane hits Florida
• Rains at place/time

• Where
• IEM, Intrade.com
• Intrade.com
• Stock options market
• Las Vegas, Betfair
• Futures market
• Weather derivatives
• Insurance company
• Weatherbill.com

15
Prediction MarketsWith Money Without
16
The Widsom of CrowdsBacked in Points

• HSX.com
• Newsfutures.com
• InklingMarkets.com
• Foresight Exchange
• CasualObserver.net
• FTPredict.com
• Yahoo!/OReilly Tech Buzz
• ProTrade.com
• StorageMarkets.com
• TheSimExchange.com
• TheWSX.com
• Alexadex, Celebdaq, Cenimar, BetBubble,
  Betocracy, CrowdIQ, MediaMammon,Owise,
  PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds
• http//www.chrisfmasse.com/3/3/markets/Play-Money
  _Prediction_Markets

17
http//betfair.com
Screen capture 2008/05/07
http//tradesports.com
Screen capture 2007/05/18
18
Example IEM 1992
Source Berg, DARPA Workshop, 2002
19
Example IEM
Source Berg, DARPA Workshop, 2002
20
Example IEM
Source Berg, DARPA Workshop, 2002
21
Does it work?
Thanks Yiling Chen

• Yes, evidence from real markets, laboratory
  experiments, and theory
• Racetrack odds beat track experts Figlewski
  1979
• Orange Juice futures improve weather forecast
  Roll 1984
• I.E.M. beat political polls 451/596 Forsythe
  1992, 1999Oliven 1995Rietz 1998Berg
  2001Pennock 2002
• HP market beat sales forecast 6/8 Plott 2000
• Sports betting markets provide accurate forecasts
  of game outcomes Gandar 1998Thaler
  1988Debnath EC03Schmidt 2002
• Laboratory experiments confirm information
  aggregationPlott 198219881997Forsythe
  1990Chen, EC01
• Theory rational expectations Grossman
  1981Lucas 1972
• Market games work Servan-Schreiber 2004Pennock
  2001

22
Prediction MarketsDoes Money Matter?
23
The Wisdom of CrowdsWith Money Without

• IEM 237 Candidates

• HSX 489 Movies

24
The Wisdom of CrowdsWith Money Without
25
Real markets vs. market games
HSX
FX, F1P6
forecast source avg log score F1P6 linear
scoring -1.84 F1P6 F1-style scoring -1.82 betting
odds -1.86 F1P6 flat scoring -2.03 F1P6 winner
scoring -2.32
26
Does money matter? Play vs real, head to head

• Experiment
• 2003 NFL Season
• ProbabilitySports.com Online football forecasting
  competition
• Contestants assess probabilities for each game
• Quadratic scoring rule
• 2,000 experts, plus
• NewsFutures (play )
• Tradesports (real )
• Used last trade prices

• Results
• Play money and real money performed similarly
• 6th and 8th respectively
• Markets beat most of the 2,000 contestants
• Average of experts came 39th (caveat)

Electronic Markets, Emile Servan-Schreiber,
Justin Wolfers, David Pennock and Brian Galebach
27
(No Transcript)
28
Does money matter? Play vs real, head to head
StatisticallyTS NFNF gtgt Avg TS gt Avg
29
Discussion

• Are incentives for virtual currency strong
  enough?
• Yes (to a degree)
• Conjecture Enough to get what people already
  know not enough to motivate independent research
• Reduced incentive for information discovery
  possibly balanced by better interpersonal
  weighting
• Statistical validations show HSX, FX, NF are
  reliable sources for forecasts
• HSX predictions gt expert predictions
• Combining sources can help

30
A Problem w/ Virtual CurrencyPrinting Money
Alice1000
Betty1000
Carol1000
31
A Problem w/ Virtual CurrencyPrinting Money
Alice5000
Betty1000
Carol1000
32
YootlesA Social Currency
Alice0
Betty0
Carol0
33
YootlesA Social Currency
I owe you 5
Alice-5
Betty0
Carol5
34
YootlesA Social Currency
I owe you 5
credit 5
credit 10
Alice-5
Betty0
Carol5
35
YootlesA Social Currency
I owe you 5
I owe you 5
credit 5
credit 10
Alice-5
Betty0
Carol5
36
YootlesA Social Currency
I owe you 5
I owe you 5
credit 5
credit 10
Alice3995
Betty0
Carol5
37
YootlesA Social Currency

• For tracking gratitude among friends
• A yootle says thanks, I owe you one

38
Combinatorial Betting
39
Combinatorics ExampleMarch Madness
40
Combinatorics ExampleMarch Madness

• Typical todayNon-combinatorial
• Team wins Rnd 1
• Team wins Tourney
• A few other props
• Everything explicit(By def, small )
• Every bet indep Ignores logical probabilistic
  relationships

• Combinatorial
• Any property
• Team wins Rnd kDuke gt UNC,NCSTACC wins 5
  games
• 2263 possible props(implicitly defined)
• 1 Bet effects related bets correctlye.g., to
  enforce logical constraints

41
ExpressivenessGetting Information

• Things you can say today
• (63 chance that) Obama wins
• GOP wins Texas
• YHOO stock gt 30 Dec 2007
• Duke wins NCAA tourney
• Things you cant say (very well) today
• Oil down, DOW up, Obama wins
• Obama wins election, if he wins OH FL
• YHOO btw 25.8 32.5 Dec 2007
• 1 seeds in NCAA tourney win more than 2 seeds

42
ExpressivenessProcessing Information

• Independent markets today
• Horse race win, place, show pools
• Stock options at different strike prices
• Every game/proposition in NCAA tourney
• Almost everything Stocks, wagers, intrade, ...
• Information flow (inference) left up to traders
• Better Let traders focus on predicting whatever
  they want, however they want Mechanism takes
  care of logical/probabilistic inference
• Another advantage Smarter budgeting

43
Market CombinatoricsPermutations

• A gt B gt C .1
• A gt C gt B .2
• B gt A gt C .1

• B gt C gt A .3
• C gt A gt B .1
• C gt B gt A .2

44
Market CombinatoricsPermutations

• D gt A gt B gt C .01
• D gt A gt C gt B .02
• D gt B gt A gt C .01
• A gt D gt B gt C .01
• A gt D gt C gt B .02
• B gt D gt A gt C .05
• A gt B gt D gt C .01
• A gt C gt D gt B .2
• B gt A gt D gt C .01
• A gt B gt C gt D .01
• A gt C gt B gt D .02
• B gt A gt C gt D .01

• D gt B gt C gt A .05
• D gt C gt A gt B .1
• D gt C gt B gt A .2
• B gt D gt C gt A .03
• C gt D gt A gt B .1
• C gt D gt B gt A .02
• B gt C gt D gt A .03
• C gt A gt D gt B .01
• C gt B gt D gt A .02
• B gt C gt D gt A .03
• C gt A gt D gt B .01
• C gt B gt D gt A .02

45
Bidding Languages

• Traders want to bet on properties of orderings,
  not explicitly on orderings more natural, more
  feasible
• A will win A will show
• A will finish in 4-7 A,C,E will finish in
  top 10
• A will beat B A,D will both beat B,C
• Buy 6 units of 1 if AgtB at price 0.4
• Supported to a limited extent at racetrack today,
  but each in different betting pools
• Want centralized auctioneer to improve liquidity
  information aggregation

46
Example

• A three-way match
• Buy 1 of 1 if AgtB for 0.7
• Buy 1 of 1 if BgtC for 0.7
• Buy 1 of 1 if CgtA for 0.7

B
A
C
47
Pair Betting

• All bets are of the form A will beat B
• Cycle with sum of prices gt k-1 gt Match(Find
  best cycle Polytime)
• Match /gt Cycle with sum of prices gt k-1
• Theorem The Matching Problem for Pair Betting is
  NP-hard (reduce from min feedback arc set)

48
Automated Market Makers
Thanks Yiling Chen

• A market maker (a.k.a. bookmaker) is a firm or
  person who is almost always willing to accept
  both buy and sell orders at some prices
• Why an institutional market maker? Liquidity!
• Without market makers, the more expressive the
  betting mechanism is the less liquid the market
  is (few exact matches)
• Illiquidity discourages trading Chicken and egg
• Subsidizes information gathering and aggregation
  Circumvents no-trade theorems
• Market makers, unlike auctioneers, bear risk.
  Thus, we desire mechanisms that can bound the
  loss of market makers
• Market scoring rules Hanson 2002, 2003, 2006
• Dynamic pari-mutuel market Pennock 2004

49
Overview Complexity Results
Permutations Permutations Permutations Boolean Boolean Boolean Taxonomy Taxonomy Taxonomy
General Pair Subset General 2-clause Restrict Tourney General Tree
Call Market NP-hard EC07 NP-hard EC07 Poly EC07 NP-hard DSS05 co-NP-complete DSS05 ? ? ?
Market Maker (LMSR) P-hard EC08 P-hard EC08 P-hard EC08 P-hard EC08 Approx STOC08 P-hard EC08 Poly STOC08 P-hard XYZ09 Poly XYZ09
50

• March Madness bet constructor
• Bet on any team to win any game
• Duke wins in Final 4
• Bet exotics
• Duke advances further than UNC
• ACC teams win at least 5
• A 1-seed will lose in 1st round

51
New Prediction Game Yoopick An Application on
Facebook
52
Catalysts

• Markets have long history of predictive accuracy
  why catching on now as tool?
• No press is bad press Policy Analysis Market
  (terror futures)
• Surowiecki's Wisdom of Crowds
• Companies
• Google, Microsoft, Yahoo! CrowdIQ, HSX,
  InklingMarkets, NewsFutures
• Press BusinessWeek, CBS News, Economist,
  NYTimes, Time, WSJ, ...http//us.newsfutures.com/
  home/articles.html

53
CFTC Role

• MayDay 2008 CFTC asks for help
• Q What to do with prediction markets?
• Yahoo!, Google entered suggestions
• Right now, the biggest prediction markets are
  overseas, academic (1), or just for fun
• CFTC may clarify, drive innovationOr not

54
Conclusion

• Prediction Marketshammer market, nail
  prediction
• Great empirical successes
• Momentum in academia and industry
• Fascinating (algorithmic) mechanism design
  questions, including combinatorial betting
• Points-paid peers produce prettygood predictions