Repeated Auction Games and Learning Dynamics in Electronic Logistics Marketplaces: Regulation through Information

1
Repeated Auction Games and Learning Dynamics in
Electronic Logistics Marketplaces Regulation
through Information
Hani S. Mahmassani University of
Maryland Potentials of Complexity Science for
Business, Governments and the Media Collegium
Budapest, August 3-5, 2006
2
Repeated Auction Games and Learning Dynamics in
Electronic Logistics Marketplaces Regulation
through Information
Hani S. Mahmassani University of
Maryland Potentials of Complexity Science for
Business, Governments and the Media Collegium
Budapest, August 3-5, 2006
3
Once upon a time, there was a physical world
4
(No Transcript)
5
Motivation

• Developments in Information and Communication
  Technologies are
• Transforming Supply Chain Operations
• Enhancing transportation service levels and
  optimizing its performance
• Introducing new ways of meeting supply (capacity)
  and demand (shippers)
• Freight Matching
• Transportation Auctions
• Supply Chain Integration Tools

6
Repeated Auction Games and Learning Dynamics in
Electronic Logistics Marketplaces Regulation
through Information
7
Presentation Outline

• Motivation
• Characteristics of Transportation Auctions
• Problem Definition
• Methodology
• Results
• Ongoing and Future Research

8
Collaborative research withMiguel Figliozzi
(former PhD Student at Maryland, now Asst.
Professor at University of Sydney) Patrick
Jaillet (MIT)
9

Range of Shipper/Carrier Procurement Structures
Vertical Integration
Long Term Contracts 3PL Services
Private Exchanges
Spot Market Brokers/Public Exchange
Private Fleet
Core Carriers
Any Carrier/Shipper
Long Term Relationships
Number of Participants
Control, Collaboration, Reliability
Savings from Better Routing Economies of
Scale/Scope
Savings from Collaboration Customized Services
10

• Illustrative Benefits of
• Market-based Procurement
• Wealth Generation

11
Comparing Transportation Market Environments

• Common Characteristics
• Stochastic arrival of shipments
• Hard Time Windows
• Simulate market under
• Different arrival rates (low to high)
• Different Time Windows Widths (short to long)
• Truth revealing second price auction

• Vertical Integration
• Assignment to fleets
• To own fleet
• One shipment at a time
• In real time
• In order of arrival

• Spot Market
• Assignment to fleets
• To best bidder
• One shipment at a time
• In real time
• In order of arrival
• (zero probability of bidding on two shipments)

Performance Total Wealth Generated, Shipments
Served, System Empty Distance
12
Vertical Integration vs. Spot Market
4 shippers and 4 Carriers
Wealth Generated Change ( increase)
13
Vertical Integration vs. Spot Market
(4 shippers and 4 Carriers Shipment Served)
14
Vertical Integration vs. Spot Market
(4 shippers and 4 Carriers Empty Distance
Change)
15
Dynamic Pricing in a Sequential Auction
Marketplace...

• The market generates a sequence of auctions
• Prices are generated as
• The outcome of carrier bids
• Predefined set of rules (auction rules)
• A Carriers behavior is expressed through his/her
  bids

16
Auction Marketplace is a useful laboratory to
gain insight into

• Carrier behavior
• Learning and Adaptation
• Effectiveness of Competitive strategies
• Impact of Information Availability on System
  Performance

17
Presentation Outline

• Motivation
• Characteristics of Transportation Auctions
• Problem Definition
• Methodology
• Results
• Ongoing and Future Research

18
What are the characteristics of transportation
auctions?

• The traded entity is a service
• Transportation services are perishable,
  non-storable commodities
• Demand and supply are geographically dispersed
  but they exchange real time information online

19
What are the characteristics of transportation
auctions?

• Group Effect value of traded item (shipment) may
  be strongly dependent upon the acquisition of
  other items (e.g. nearby shipments)
• Network Effect value of a shipment is related to
  the current spatial and temporal deployment of
  the fleet.
• Uncertainty
• demand/supply over time and space
• prices

20
Sources of Complexity

• Multiple interacting agents with multiple
  conflicting objectives
• Modeling agents bounded rationality and learning
• Demand spatial and temporal stochasticity
• Uncertainties about a shipment value and cost
• Fleet management complexities (routing, time
  windows, penalties, etc.)
• New class of problems created by different market
  designs and levels of information

21
Presentation Outline

• Motivation
• Characteristics of Transportation Auctions
• Problem Definition
• Research Methodology
• Results
• Ongoing and Future Research

22
Problem Context

• Sequential Auction Market Environment
• Stochastic arrival of non-identical shipments
• Sequential auction of arriving shipments
• Bidding is done one shipment at a time, in order
  of arrival
• Carriers objective is to maximize expected
  profits while managing the fleet to satisfy
  service quality constraints (time windows)
• The principal operating costs are proportional
  to the shipment haul-length and the empty
  distance required
• Carrier costs and capacity affected by history of
  bidding and shipment assignment decisions

23
Problem Categorization

• Two Layers of Allocations
• Auction Shippers ? Bidders
• Pricing of resources
• Profit Maximization problem
• Strategic Problem
• Fleet Management Shipments ? Trucks
• Allocation of own resources
• Cost Minimization problem
• Non-strategic problem
• The joint bidding/fleet management problem is
  highly complex

24
Game Formulation

• A sequential auction as a dynamic game of
  imperfect information
• dynamic carriers face each other at different
  stages
• imperfect information carriers are uncertain
  about competitors private information (what
  affects competitors shipment cost)
• Stages are identified with shipment arrival
  epochs
• A carrier has full knowledge about his fleet
  status (vehicles and shipments) and technology
• A carrier has uncertainty about competitors
  fleet status, technology, or bidding function

25
Finding a Bidding Policy

• In auctions, profits are highly dependent on the
  quality of the bidding policy.
• A bidding policy is a function produces a bid
  value using information about
• the state of the carrier,
• the characteristics of the shipment for auction,
• the marginal cost of serving the shipment,
• auction type, and beliefs about the competitors
  and environment

26
Finding a Bidding Policy

• Problem complexity generally precludes finding a
  policy that optimizes the entire
  auction/assignment problem.
• Each auction provides opportunity for carriers to
  learn about
• The environment
• Other players strategies
• Learning potential is dependent on information
  disclosed after each auction

27
Information Levels

• The information revealed after each auction can
  influence the nature and rate of the process by
  which carriers learn about the game and their
  competitors behavior.
• Information includes
• Actions (bids) placed.
• Number of players (carriers) participating
• Links (name) between carriers and bids
• Individual characteristics of carriers (e.g.
  fleet size)
• Payoffs received
• Knowledge about who knows what, information
  asymmetries, or shared knowledge about previous
  items.

28
Information Levels

• Define TWO information levels
• maximum information environment, all the above
  information is revealed.
• minimum information environment where no
  information is revealed.
• These two extremes approximate two realistic
  situations
• Maximum information would correspond to a real
  time internet auction where all auction
  information is accessed by participants.
• Minimum information would correspond to a shipper
  telephoning carriers for a quote, and calling
  back only the selected carrier.

29
Learning in Different Environments

• Minimum information setting
• Genetic Algorithms
• Reinforcement Learning
• Maximum information setting
• Fictitious Play
• Machine Learning
• Rationalizable and Machine Learning
• Rule Learning
• Rules of Thumb Learning (e.g. Tit for Tat)

30
Carriers Decisions

• Strategic decisions investment of resources, for
  the purpose of learning about or influencing
  competitors, to improve profits by manipulating
  future auction outcomes.
• Identifying decisions are characterized by
  attempts to identify or discover a competitors
  behavior.
• Signals are decisions that aim to establish a
  reputation or status for the carrier.
• Operating decisions decisions that are not
  strategic but aim to improve a carriers profit
  level (e.g. rerouting of the fleet after a
  successful bid)

31
Bounded Rationality

• Carriers can analyze with different degrees of
  sophistication (bounded rationality) the history
  of play and estimate the possible future
  consequences of current actions.
• Research in the area of learning in games is
  actively seeking to explain how agents acquire,
  process, evaluate or search for information.

32
Bounded Rationality

• Cognitive and computational limitations can be
  evidenced in
• Identification the carrier has limited ability
  to discover competitors behavioral types, which
  may require complex econometric techniques
• Signaling limited ability to read or send
  signals that convey a reputation
• Memory limited ability to record and keep past
  outcome information or memory to simulate all
  future possible paths in the decision tree
• Optimization even if carriers could identify
  competitors behavior, their ability to formulate
  and solve stochastic optimization problems is
  likely limited.

33
Presentation Outline

• Introduction
• Characteristics of Transportation Auctions
• Problem Definition
• Research Methodology
• Results
• Ongoing and Future Research

34
Case Study Myopic Carrier Learning in Second
Price Auctions

• Study the impact of different learning techniques
  on
• Carriers
• Profits
• Market share
• Under different market settings
• Minimum Information
• Maximum Information

• Shippers
• Consumer Surplus
• Number Shipments Served

• Low Arrival Rate (uncongested)
• High Arrival Rate (congested)

35
Research Methodology

• Define Auction Type Second Price Auction
• DEFINITION (reverse auction)
• Carrier with lowest bid wins item
• Winner gets paid second lowest bid
• Rest of bidders do not pay or receive anything
• PROPERTIES (one shot auction - Vickrey 1961)
• Equilibrium strategies are truth-revealing and
  dominant strategies bid true marginal cost
• They do not require gathering or analysis of
  information about the competitors situation
• Leads to complete economic efficiency, the bidder
  with the lowest cost wins

36
Problems with Second Price Auctions

• Rothkopf, Teisberg, and Kahn (1990, JPE)
• Auctioneer may cheat in the auction
• Vulnerability to bidder collusion
• Revelation of private information
• Sandholm (2000, IJEC)
• Complexity of bidding looking at future sequence
  of arrivals introduce speculation about future
  bids of other bidders
• Untruthful bidding with risk averse bidders
• Marginal Cost Bidding (Krishna, 2002)
• Not necessarily Optimal in sequential Auctions
• No known equilibrium for sequential auctions
  where bidders have multi-unit demand curves

37
Finding an Optimal policy...

• Limited to myopic bidding
• Will not consider impact of bidding on
  competitors future behavior
• Will not consider impact of bidding on future
  service costs
• Limited to finding the best constant marginal
  cost factor c such that
• bid c x marginal cost

38
Finding an Optimal policy...

• Compare Learning Strategies
• Reinforcement Learning
• Tit for Tat
• And
• Marginal Cost Bidding ( c 1)

39
Reinforcement Learning

• An agent chooses an action with a probability
  that is directly proportional to the profit that
  such action has achieved in the past
• Initially the agent starts with positive uniform
  profits over all possible marginal cost factors
• As each action (bid) is played, the agent updates
  the profit level with the payoff obtained
• Over time, profit levels will converge (if
  facing a stationary environment)
• This learning method can be utilized under
  minimum or maximum information settings

go to referring slide
40
Reinforcement Learning (ctd.)

• A a1, ... ,an set of available actions (bids)
• r(ai) average reward obtained using action ai
  in the past (includes both won and lost bids)
• P (ai) probability of playing action ai
• P (ai) r(ai) / Si r(ai) i?A

go to referring slide
41
Tit for Tat

• Tit for Tat is more an adaptive rule of thumb
  than a learning mechanism
• It is a robust strategy in many strategic
  situations
• This carrier roughly imitates what his opponent
  is doing
• With two carriers, A and T, where carrier T plays
  Tit for Tat, this rule of thumb can be defined
  as
• Carrier T computes the average bid value of
  carrier A over the last T auctions, called â
• Carrier T computes his own marginal cost average
  over the last T auctions, called c
• Carrier T obtains a â / c,
• Ts next bid will be equal to bid a x
  marginal cost

42
Behavioral Assumptions and Rules

• Carriers
• Non-cooperative carriers
• Preference over game outcomes with highest
  expected return, risk neutral
• Myopic strategies
• Shippers
• Shipper selects carrier with lowest bid
• Shipper does not cheat

43
Other Market Settings

• 2 Carriers
• Geographic Area 1 1 square space
• Shipment Origin and Destination ? Uniformly
  distributed over space
• Earliest Pick Up Time arrival time
• Latest Pick Up Time arrival time Time window
  length (2 units of time uniform0,2 )
• Fleet size 12 vehicles (constant) serving the
  market
• The reservation price of the buyer (shipper) is
  distributed uniform 1.4,1.5
• ? 0.25 arrivals/unit time/truck (not congested)
• ? 1.00 arrivals/unit time/truck (congested)
• Results obtained with 10 iterations of 10,000
  arrivals each

44
Presentation Outline

• Introduction
• Characteristics of Transportation Auctions
• Problem Definition
• Methodology
• Results
• Ongoing and Future Research

45
Optimality of Bidding Marginal Cost with Low
Arrival Rates (AR3)

• Carrier MC bids always marginal cost
• Carrier D bids marginal cost multiplied by c
• Highest profits when c 1

46
Reinforcement Learning

• Carriers discover that c 1 provides the highest
  profits among all possible factors

47
Learning is not Free

• Comparing Shipper Surplus and Rejected Shipments
  between carriers playing Reinforcement Learning
  (RL) and carriers bidding Marginal Cost (MC)

48
Learning is not competitive...

• Comparing Shipments Won and Profits when a
• RL carrier competes against a MC (c1) carrier

49
Tit for Tat is a Robust Strategy

• Maximum Information Setting
• Tit for Tat competing with a RL and MC carriers
  (profits)
• Tit for Tat successfully imitates competitor

50
Too much information could be a problem...

• If the leader becomes aware of his leadership,
  it is a dominating strategy to rise prices
• Graphs compare profits for MC carrier and Tit for
  Tat carrier when the leader goes from c1 to c2

51
Conclusions

• Even with minimum information, Learning is
  possible (e.g. convergence towards mc bidding)
• Learning is expensive for both carriers and
  shippers
• Carriers suffer hard against competitors that
  have already found optimal policies
• Shippers do pay for learning when all carriers
  are learning
• Higher prices
• Fewer shipments served
• Therefore, market setting should be such that
  learning duration is minimized

52
Conclusions

• Maximum information settings allow a large array
  of possible new behaviors
• Tit for Tat or imitation is possible, typical
  market with a leader and a follower
• From a carrier point of view, Tit for Tat is
  robust
• From shippers point of view, Tit for Tat is
  good as long as the leader follows a
  competitive bidding policy
• Problem with too much information
• If the leader becomes aware of his leadership,
  it is a dominating strategy to raise prices
• The follower gladly follows suit since his
  profits also rise

53

• Incentive Compatibility Issues

54
Incentive Compatibility

• Truth revealing second price auction market has a
  wealth creation potential
• Can a truth revealing market be sustained?
• Experiment compare the performance of a truth
  revealing carrier and a NON-truth revealing
  carrier
• One carrier uses a bidding factor ?1
• The other carrier bids his marginal cost

55
Ongoing and Future Research

• Develop more sophisticated strategies that do
  take into account future consequences of current
  bid on
• Players own future costs
• Players own future revenues
• Develop strategies that use marketplace
  information to identify competitors behavior and
  manipulate future outcomes
• Study how market performance is affected by
  varying
• number of carriers
• auction mechanisms
• Information disclosed
• Develop discrete choice models of competitor
  behavior under different observation and
  information conditions

56

• QUESTIONS ?

57
Game Formulation

• The Truck-Load Procurement Market (TLPM)
  formulation differs from other auction
  formulations in several respects
• (a) items auctioned (shipments) are
  multi-attributed
• (b) costs are functions of carriers status and
  vehicle routing technologies
• (c) history and fleet management decisions affect
  future cost probability distributions.
• (d) capacity constraints are linked to private
  information and shipment characteristics
• (e) bidding strategies are dependent on public
  and private history
• (f) timing of auctions is important
• (e) it is an online sequential auction.

58
Notation
carriers competing, each carrier
(The set of carriers)
the set of auction announcement epochs is

the set of arriving shipments is
arrives
represents the time when shipment
(set of shipments arriving after )
each carrier simultaneously bids a monetary amount
.
59
Notation
public information generated after auction for
information publicly known before bidding for
shipment
private information for each carrier at time
carrier status before bidding (shipments fleet)
assignment function
cost function
conditional probability about opponents type
60
Notation
bidding function
competitors bidding functions
auction assignment function
probabilities of winning shipment
auction expected payment function
expected profit for shipment
61
Online Equilibrium
probability space of arrivals and shipment
characteristics
Bidding Functions Equilibrium
62
Relaxing Rationality

• In a bounded rational model, a carrier faces two
  basic types of uncertainties regarding the
  competition
• an uncertainty relative to the competitors
  private information
• an uncertainty relative to the competitors
  bounded rationality type or bidding function
• These uncertainties can be combined into a
  price function
• Stationary case

63
Price Problem Formulation1st price Auction
64
Price Problem Formulation2nd price Auction
return