Prediction Markets and the Wisdom of Crowds

1
Prediction Markets and the Wisdom of Crowds

• David M. Pennock, Yahoo! Research
• Yiling Chen, Lance Fortnow, Evdokia Nikolova,
  Daniel Reeves

2
Hammers and Nails
Tool Prediction
Application Markets
e.g.
3
Hammers and Nails
Tool Markets
Application Prediction
4
A Prediction Market

• Take a random variable, e.g.
• Turn it into a financial instrument payoff
  realized value of variable

Bird Flu Outbreak US 2007?(Y/N)
I am entitled to
Bird FluUS 07
Bird FluUS 07
1 if
0 if
5
http//intrade.com
Screen capture 2007/05/18
6
Outline

• The Wisdom of Crowds
• The Wisdom of Markets
• Prediction Markets Examples Research
• Mechanism Design
• Betting on Permutations
• Dynamic Parimutuel Market
• The Coming Convergence ofStats Mechanism Design

Story
Survey
Research
Opinion
7
A WOC Story
Story
1/7
Survey
Research
Opinion

• ProbabilitySports.com
• Thousands of probability judgments for sporting
  events
• Alice Jets 67 chance to beat Patriots
• Bob Jets 48 chance to beat Patriots
• Carol, Don, Ellen, Frank, ...
• Reward Quadratic scoring ruleBest probability
  judgments maximize expected score

8
Individuals

• Most individuals are poor predictors
• 2005 NFL Season
• Best 3747 points
• Average -944 Median -275
• 1,298 out of 2,231 scored below zero(takes
  work!)

9
Individuals

• Poorly calibrated (too extreme)
• Teams given lt 20 chance actually won 30 of the
  time
• Teams given gt 80 chance actually won 60 of the
  time

10
The Crowd

• Create a crowd predictor by simply averaging
  everyones probabilities
• Crowd 1/n(Alice Bob Carol ... )
• 2005 Crowd scored 3371 points(7th out of 2231)
  !
• Wisdom of fools Create a predictor by averaging
  everyone who scored below zero
• 2717 points (62nd place) !
• (the best fool finished in 934th place)

11
The Crowd How Big?
Morehttp//blog.oddhead.com/2007/01/04/the-wisdo
m-of-the-probabilitysports-crowd/http//www.overc
omingbias.com/2007/02/how_and_when_to.html
12
Can We Do Better? ML/Stats
Dani et al. UAI 2006

• Maybe Not
• CS experts algorithms
• Other expert weights
• Calibrated experts
• Other averaging fns (geo mean, RMS, power means,
  mean of odds, ...)
• Machine learning (NB, SVM, LR, DT, ...)
• Maybe So
• Bayesian modeling EM
• Nearest neighbor (multi-year)

13
Can we do better? Markets
14
Prediction MarketsExamples Research
15
http//www.biz.uiowa.edu/iem
16
Example IEM 1992
Source Berg, DARPA Workshop, 2002
17
Example IEM
Source Berg, DARPA Workshop, 2002
18
Example IEM
Source Berg, DARPA Workshop, 2002
19
http//tradesports.com
http//intrade.com
Screen capture 2007/05/18
20
http//www.wsex.com/
http//www.hedgestreet.com/
Screen capture 2007/05/18
Screen capture 2007/05/18
21
Play moneyReal predictions
http//www.hsx.com/
22
http//us.newsfutures.com/
http//www.ideosphere.com
Cancercuredby 2010
Machine Gochampionby 2020
23
Yahoo!/OReilly Tech Buzz Game
http//buzz.research.yahoo.com/
24
More Prediction Market Games

• BizPredict.com
• CasualObserver.net
• FTPredict.com
• InklingMarkets.com
• ProTrade.com
• StorageMarkets.com
• TheSimExchange.com
• TheWSX.com
• Alexadex, Celebdaq, Cenimar, BetBubble,
  Betocracy, CrowdIQ, MediaMammon,Owise,
  PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds
• http//www.chrisfmasse.com/3/3/markets/Play-Money
  _Prediction_Markets

25
Does it work?Yes...

• Evidence from real markets, laboratory
  experiments, and theory indicate that markets are
  good at gathering information from many sources
  and combining it appropriately e.g.
• Markets like the Iowa Electronic Market predict
  election outcomes better than pollsForsythe
  1992, 1999Oliven 1995Rietz 1998Berg
  2001Pennock 2002
• Futures and options markets rapidly incorporate
  information, providing accurate forecasts of
  their underlying commodities/securitiesSherrick
  1996Jackwerth 1996Figlewski 1979Roll
  1984Hayek 1945
• Sports betting markets provide accurate forecasts
  of game outcomes Gandar 1998Thaler
  1988Debnath EC03Schmidt 2002

26
Does it work?Yes...

• E.g. (contd)
• Laboratory experiments confirm information
  aggregationPlott 198219881997Forsythe
  1990Chen, EC-2001
• And field tests Plott 2002
• Theoretical underpinnings rational
  expectationsGrossman 1981Lucas 1972
• Procedural explanation agents learn from
  pricesHanson 1998Mckelvey 1986Mckelvey
  1990Nielsen 1990
• Proposals to use information markets to help
  science Hanson 1995, policymakers, decision
  makers Hanson 1999, government Hanson 2002,
  military DARPA FutureMAP, PAM
• Even market games work! Servan-Schreiber
  2004Pennock 2001

27
Catalysts

• Markets have long history of predictive accuracy
  why catching on now as tool?
• No press is bad press Policy Analysis Market
  (terror futures)
• Surowiecki's Wisdom of Crowds
• Companies
• Google, Microsoft, Yahoo! CrowdIQ, HSX,
  InklingMarkets, NewsFutures
• Press BusinessWeek, CBS News, Economist,
  NYTimes, Time, WSJ, ...http//us.newsfutures.com/
  home/articles.html

28
Mechanism Design IBetting on Permutations
29
Mech Design for Prediction
Financial Markets Prediction Markets
Primary Social welfare (trade)Hedging risk Information aggregation
Secondary Information aggregation Social welfare (trade)Hedging risk
30
Mech Design for Prediction

• Standard Properties
• Efficiency
• Inidiv. rationality
• Budget balance
• Revenue
• Comp. complexity
• Equilibrium
• General, Nash, ...

• PM Properties
• 1 Info aggregation
• Expressiveness
• Liquidity
• Bounded budget
• Indiv. rationality
• Comp. complexity
• Equilibrium
• Rational expectations

Competes withexperts, scoringrules,
opinionpools, ML/stats,polls, Delphi
31
Predicting Permutations

• Predict the ordering of a set of statistics
• Horse race finishing times
• Number of votes for several candidates
• Daily stock price changes
• NFL Football quarterback passing yards
• Any ordinal prediction
• Chen, Fortnow, Nikolova, Pennock, EC07

32
Market CombinatoricsPermutations

• A gt B gt C .1
• A gt C gt B .2
• B gt A gt C .1

• B gt C gt A .3
• C gt A gt B .1
• C gt B gt A .2

33
Market CombinatoricsPermutations

• D gt A gt B gt C .01
• D gt A gt C gt B .02
• D gt B gt A gt C .01
• A gt D gt B gt C .01
• A gt D gt C gt B .02
• B gt D gt A gt C .05
• A gt B gt D gt C .01
• A gt C gt D gt B .2
• B gt A gt D gt C .01
• A gt B gt C gt D .01
• A gt C gt B gt D .02
• B gt A gt C gt D .01

• D gt B gt C gt A .05
• D gt C gt A gt B .1
• D gt C gt B gt A .2
• B gt D gt C gt A .03
• C gt D gt A gt B .1
• C gt D gt B gt A .02
• B gt C gt D gt A .03
• C gt A gt D gt B .01
• C gt B gt D gt A .02
• B gt C gt D gt A .03
• C gt A gt D gt B .01
• C gt B gt D gt A .02

34
Bidding Languages

• Traders want to bet on properties of orderings,
  not explicitly on orderings more natural, more
  feasible
• A will win A will show
• A will finish in 4-7 A,C,E will finish in
  top 10
• A will beat B A,D will both beat B,C
• Buy 6 units of 1 if AgtB at price 0.4
• Supported to a limited extent at racetrack today,
  but each in different betting pools
• Want centralized auctioneer to improve liquidity
  information aggregation

35
Auctioneer Problem

• Auctioneers goalAccept orders with
  non-negative worst-case loss (auctioneer never
  loses money)
• The Matching Problem
• Formulated as LP
• Generalization Market Maker ProblemAccept
  orders with bounded worst-case loss (auctioneer
  never loses more than b dollars)

36
Example

• A three-way match
• Buy 1 of 1 if AgtB for 0.7
• Buy 1 of 1 if BgtC for 0.7
• Buy 1 of 1 if CgtA for 0.7

B
A
C
37
Pair Betting

• All bets are of the form A will beat B
• Cycle with sum of prices gt k-1 gt Match(Find
  best cycle Polytime)
• Match /gt Cycle with sum of prices gt k-1
• Theorem The Matching Problem for Pair Betting is
  NP-hard (reduce from min feedback arc set)

38
Subset Betting

• All bets are of the form
• A will finish in positions 3-7, or
• A will finish in positions 1,3, or 10, or
• A, D, or F will finish in position 2
• Theorem The Matching Problem for Subset Betting
  is polytime (LP maximum matching separation
  oracle)

39
Mechanism Design IIDynamic Parimutuel Market
40
Automated Market Makers
Thanks Yiling Chen

• A market maker (a.k.a. bookmaker) is a firm or
  person who is almost always willing to accept
  both buy and sell orders at some prices
• Why an institutional market maker? Liquidity!
• Without market makers, the more expressive the
  betting mechanism is the less liquid the market
  is (few exact matches)
• Illiquidity discourages trading Chicken and egg
• Subsidizes information gathering and aggregation
  Circumvents no-trade theorems
• Market makers, unlike auctioneers, bear risk.
  Thus, we desire mechanisms that can bound the
  loss of market makers
• Market scoring rules Hanson 2002, 2003, 2006
• Dynamic pari-mutuel market Pennock 2004

41
Automated Market Makers
Thanks Yiling Chen

• n disjoint and exhaustive outcomes
• Market maker maintain vector Q of outstanding
  shares
• Market maker maintains a cost function C(Q)
  recording total amount spent by traders
• To buy ?Q shares trader pays C(Q ?Q) C(Q) to
  the market maker Negative payment receive
  money
• Instantaneous price functions are
• At the beginning of the market, the market maker
  sets the initial Q0, hence subsidizes the market
  with C(Q0).
• At the end of the market, C(Qf) is the total
  money collected in the market. It is the maximum
  amount that the MM will pay out.

42
What is a pari-mutuel market?

• E.g. horse racetrack style wagering
• Two outcomes A B
• Wagers

43
What is a pari-mutuel market?
A
B

• E.g. horse racetrack style wagering
• Two outcomes A B
• Wagers

?
44
What is a pari-mutuel market?
A
B

• E.g. horse racetrack style wagering
• Two outcomes A B
• Wagers

?
45
What is a pari-mutuel market?

• Before outcome is revealed, odds are reported,
  or the amount you would win per dollar if the
  betting ended now
• Horse A 1.2 for 1 Horse B 25 for 1 etc.
• Strong incentive to wait
• payoff determined by final odds every is same
• Should wait for best info on outcome, odds
• ? No continuous information aggregation
• ? No notion of buy low, sell high no cash-out

46
Pari-Mutuel MarketBasic idea
1
1
1
1
1
1
1
1
1
1
1
1
47
Dynamic Parimutuel Market
C(1,2)2.2
.82
C(2,3)3.6
.78
C(2,2)2.8
.59
.87
.3
C(2,4)4.5
.4
C(3,8)8.5
.91
.49
.94
C(4,8)8.9
C(2,5)5.4
.96
C(5,8)9.4
0.97
C(2,6)6.3
C(2,7)7.3
C(2,8)8.2
48
Share-ratio price function

• One can view DPM as a market maker
• Cost Function
• Price Function
• Properties
• No arbitrage
• pricei/pricej qi/qj
• pricei lt 1
• payoff if right C(Qfinal)/qo gt 1

49
Open QuestionsCombinatorial Betting

• Usual hunt Are there natural, useful, expressive
  bidding languages (for permutations, Boolean,
  other) that admit polynomial time matching?
• Are there good heuristic matching algorithms
  (think WalkSAT for matching) logical reduction?
• How can we divide the surplus?
• What is the complexity of incremental matching?

50
Open QuestionsAutomated Market Makers

• For every bidding language with polytime
  matching, does there exist a polytime MSR market
  maker?
• The automated MM algorithms are online
  algorithms Are there other online MM algorithms
  that trade more for same loss bound?

51
Hammers and Nails
Tool Prediction
Tool Markets
Application Markets
Application Prediction
52
Coming ConvergenceStats and Mechanism Design
Mechanism(Rules) e.g. Auction,Exchange, ...
53
Convergence Advertising
54
ML Inner Loop

• Optimal allocation (ad-user match) depends on
  bid, Eclicks, Esales, relevance, ad,
  advertiser, user, context (page, history), ...
• Expectations must be learned
• Learning in dynamic setting requires
  exploration/exploitation tradeoff
• Mechanism design must factor all this in!
  Nontrivial.

55
Adversarial Machine Learning

• Learning in a game theoretic environment
• Spam!
• Click fraud
• Shilling

56
Incentive-Centered Design
Technology Manipulation
Email Spam
Web/Altavista Keyword spam
Web/Google Link spam
Reviews/recommendations Shilling
Sponsored search Click spam
Blogs Comment/trackback spam
Tags (Flickr) Tag spam
Social Aggregators (Digg) Shilling
Semantic Web ??

• This is not vandalism! All based on economic
  incentives
• Havent we learned? Needed Incentive-centered
  design
• See UM SI http//www.si.umich.edu/research/area.h
  tm?AreaID4
• Yahoo! Research Microeconomics Sociology

57
Conclusion

• Market Predictionhammer prediction, nail
  market
• Prediction Marketshammer market, nail
  prediction
• Great empirical successes
• Momentum in academia and industry
• Fascinating (algorithmic) mech design questions
• Convergence Happening
• hammer nail prediction market
• Prediction in inner loop of mechanism design
• Nowhere more clear than online advertising