Construction of Markov Arrival Processes

1

• Construction of Markov Arrival Processes
• Qi-Ming HE
• Department of Industrial Engineering
  
• Dalhousie University, Canada B3J 2X4
• Presented at Department of Statistics and
  Actuariy Science
• University of Western Ontario, December 3, 2009

2

• A system provides service to customers
• How to model the arrival process mathematically?
• such that the system can be investigated
  analytically?
• and system descriptors can be computed
  numerically?

3
Outline

• Introduction
• Exponential Distribution and Poisson Process
• Construction of CTMCs
• Construction of MAPs and BMAPs
• Construction of MMAPs
• An Example

4
1. Introduction

• A brief history of Markov arrival processes
  (MAPs)
• 1979 MAPs introduced by Marcel Neuts.
• 1979 1990 Used in the study of queueing models
  (e.g., N/G/1)
• 1991 Named as MAP and matrix representation
  simplified.
• 1996 Marked MAPs introduced and used in the
  study of queueing models.
• 1990 present Used to model arrival processes
  of queueing, reliability, inventory, supply
  chain, risk and insurance, and telecommunications
  systems.

5
1. Introduction

• A list of references
• Asmussen, S. and G. Koole (1993), Marked point
  processes as limits of Markovian arrival streams,
  J. Appl. Probab., Vol 30, 365-372.
• HE, Qi-Ming (1996), Queues with marked customers,
  Adv. Appl. Prob., Vol 28, 567-587.
• Lucantoni, D.M., K. Meier-Hellstern and M.F.
  Neuts (1990). A single-server queue with server
  vacations and a class of non-renewal arrival
  processes. Advances in Applied Probability, 22,
  676-705.
• Lucantoni, D.M. (1991), New results on the single
  server queue with a batch Markovian arrival
  process, Stochastic Models, Vol 7, 1-46.
• Neuts, M.F. (1979), A versatile Markovian point
  process, Journal of Applied Probability, Vol. 16,
  764-779.
• Ramaswami, V. (1980), The N/G/1 queue and its
  detailed analysis, Adv. Appl. Prob., 12, 222-61.
• Rudemo, M. (1973) Point processes generated by
  transitions of Markov chains. Adv. Appl. Prob. 5,
  262-286.

6
1. Introduction (continued)

• Advantages of MAPs
• Versatility approximating any (batch) arrival
  processes
• Markovian property Markov structure
• Tractability analytically and numerically
• Simplicity to learn and to use.

7
1. Introduction (continued)

• Key idea
• Use a supplementary variable phase to
  generate partial memoryless, which leads to a
  Markov structure for the system.

8
2. Exponential and Poisson

• Exponential distribution
• Definition 2.1 A nonnegative random variable X
  has an exponential distribution if its
  probability distribution function is given by
• where ? is a positive real number. We call X an
  exponential distribution with parameter ?.

9
2. Exponential and Poisson (continued)

• Exponential distribution Properties
• Memoryless property
• Assume that X1, X2, and X3 are three independent
  exponential random variables with parameters ?1,
  ?2, and ?3, respectively. We have, for small t,

10
2. Exponential and Poisson (continued)

• Exponential distribution Properties (continued)
• Assume that Xj, 1 ? j ? n are independent
  exponential random variables with parameters ?j,
  1 ? j ? n, respectively. Then X minX1, ,
  Xn is exponentially distributed with parameter
  ?1?n.

11
2. Exponential and Poisson (continued)

• Exponential distribution Properties (continued)
• Assume that Xj, 1 ? j ? n are independent
  exponential random variables with parameters ?j,
  1 ? j ? n, respectively. Then PX1 minX1, ,
  Xn ?1/(?1?n).
• Assume that Xn, n ? 0 are independent
  exponential random variables with the same
  parameter ?. Assume that N, independent of Xn,
  n ? 0, has a geometric distribution with
  parameter p on positive integers 1, 2, , i.e.,
  PNn pn1(1p), n ? 1. Define
• Then Y has an exponential distribution
  with parameter (1p)?.

12
2. Exponential and Poisson (continued)

• Poisson process
• Definition 2.2 A counting process N(t), t ? 0
  is called a Poisson process if , for n?0 and t?0,
  where X1, X2, , Xn, are independent
  exponential random variables with parameter ?.

13
2. Exponential and Poisson (continued)

• Poisson process Properties
• N(0) 0.
• E(N(t)) ?t. Parameter ? is the average number
  of events per unit time.
• Let Y be the time elapsed until the first event
  after time t. Then Y has an exponential
  distribution with parameter ?. This is called
  the memoryless property of Poisson processes.

14
3. Construction of CTMCs

• Construction of CTMCs The construction is based
  on Poisson processes. Parameters
• ?i, 1 ? i ? m be nonnegative numbers with a
  unit sum (i.e., ?1?m 1) For the
  determination of the initial phase.
• qi,j, 1 ? i ? j ? m nonnegative real numbers,
  and m a finite positive integer (m?2). Assume
  qi,1 qi,i1 qi,i1 qi,m gt 0, for 1 ? i ?
  m.

15
3. Construction of CTMCs (continued)

• Definition 3.1 A stochastic process I(t), t?0
  on phases 1, 2, , m is defined as follows
• Define m(m1) independent Poisson processes with
  parameters qi,j, 1 ? i ? j ? m. If qi,j 0,
  the corresponding Poisson process has no event at
  all.
• Determine I(0) by probability distribution ?i, 1
  ? i ? m.
• At time t ? 0, if I(t) i, then I(t) stays in
  phase i until the first event occurs in the m1
  Poisson processes corresponding to qi,j, 1 ? j ?
  m, j ? i, for 1 ? i ? m. If the event occurs at
  s (gt t) and it comes from the Poisson process
  corresponding to qi,j, the process transits from
  phase i to phase j at time s, i.e., I(s) i and
  I(s) j.

16
3. Constructions of CTMCs (continued)

• A sample path of a CTMC with m 3

17
3. Constructions of CTMCs (continued)

• 3.1 Properties of the stochastic process I(t),
  t?0
• Lemma 3.1 For the process I(t), t?0, we have
• The sojourn time of I(t), t?0 in phase i has an
  exponential distribution with parameter qi,i ?
  qi,1 qi,i-1 qi,i1 qi,m gt 0, for 1
  ? i ? m and
• The probability that the next phase is j is given
  by ri,j ? qi,j /(qi,i), given that the current
  phase is i, for 1 ? i ? j ? m.
• Theorem 3.2 The stochastic process I(t), t?0
  is a CTMC with m phase. The infinitesimal
  generator of CTMC I(t), t?0 is an m?m matrix Q
  (qi,j).

18
3. Constructions of CTMCs (continued)

• 3.1 Properties of the stochastic process I(t),
  t?0
• Define pi,j(t) PI(t)j I(0)i, for t ? 0
  and 1 ? i, j ? m. Denote by P(t) (pi,j(t)),
  an m?m matrix, for t?0. Here are the well-known
  Kolmogorov differential equations.
• Theorem 3.3 For the CTMC I(t), t?0, we have
  P?(t) P(t)Q QP(t), for tgt0, and P(0) I,
  where I is the identity matrix.
• Note that the (mathematically rigorous) proof of
  Theorem 3.3 is only based on the properties of
  exponential distributions and Poisson processes
  given in Section 2.

19
4. Construction of MAPs and BMAPs

• Construction of MAPs The construction is based
  on Poisson processes. Parameters
• ?i, 1 ? i ? m be nonnegative numbers with a
  unit sum (i.e., ?1?m 1) For the
  determination of the initial phase.
• d0,(i,j), 1 ? i ? j ? m and d1,(i,j), 1 ? i, j
  ? m are nonnegative numbers, and m is a finite
  positive integer. Assume d0,(i,i) ? d0,(i,1)
  d0,(i, i1)d0,(i,i1) d0,(i,m) d1,(i,1)
  d1,(i,i) d1,(i,m) gt 0, for 1 ? i ? m.

20
4. Construction of MAPs and BMAPs (cont.)

• Definition 4.1 We define a stochastic process
  (N(t), I(t)), t?0 as follows.
• Define m(2m1) independent Poisson processes with
  parameters d0,(i,j), 1 ? i ? j ? m and
  d1,(i,j), 1 ? i, j ? m. If d0,(i,j) 0 or
  d1,(i,j) 0, the corresponding Poisson process
  has no event.
• Determine I(0) by the probability distribution
  ?i, 1 ? i ? m. Set N(0) 0.
• If I(t) i, for 1 ? i ? m, I(t) and N(t) remain
  the same until the first event occurs in the 2m1
  Poisson processes corresponding to d0,(i,j), 1 ?
  j ? m, j ? i and d1,(i,j), 1 ? j ? m. If the
  next event comes from the Poisson process
  corresponding to d0,(i,j), the variable I(t)
  changes from phase i to phase j and N(t) does not
  change at the epoch, for 1 ? j ? m, j ? i If the
  next event comes from the Poisson process
  corresponding to d1,(i,j), the phase variable
  I(t) transits from phase i to phase j and N(t) is
  increased by one at the epoch, i.e., an arrival
  occurs, for 1 ? j ? m.

21
4. Construction of MAPs and BMAPs (cont.)

• A sample path of an MAP with m 3

22
4. Construction of MAPs and BMAPs (cont.)

• Properties of MAP (N(t), I(t)), t?0
• Lemma 4.1 For the process (N(t), I(t)), t?0,
  we have
• The sojourn time of (N(t), I(t)), t?0 in state
  (n, i) has an exponential distribution with
  parameter d0,(i,i), for 1 ? i ? m
• The probability that the next phase is j and no
  arrival at the transition epoch is given by
  p0,(i,j) ? d0,(i,j)/(d0,(i,i)), given that the
  current state is (n, i), for 1 ? i ? j ? m and
  n?0.
• The probability that the next phase is j and an
  arrival occurs at the transition epoch is given
  by p1,(i,j) ?
  d1,(i,j)/(d0,(i,i)), given that the current
  state is (n, i), for 1 ? i, j ? m and n?0.

23
4. Construction of MAPs and BMAPs (cont.)

• Properties of the MAP (N(t), I(t)), t?0
• Theorem 4.2 The stochastic process I(t), t?0
  is a continuous time Markov chain with an
  infinitesimal generator D, where D0 (d0,(i,j)),
  D1 (d1,(i,j)), and D D0 D1, three m?m
  matrices.
• Theorem 4.3 The stochastic process (N(t),
  I(t)), t?0 is a continuous time Markov chain
  with an infinitesimal generator
• The pair (D0, D1) is called a matrix
  representation of the MAP.

24
4. Construction of MAPs and BMAPs (cont.)

• Properties of the MAP (N(t), I(t)), t?0
• Define pi,j(n, t) PN(t) n, I(t)j I(0)i,
  for 1 ? i, j ? m, and n ? 0,
• Denote by P(n, t) (pi,j(n, t)), an m?m matrix,
  for t?0 and n?0.
• Then P?(0, t) P(0, t)D0, P(0, 0) I, P?(n, t)
  P(n, t)D0 P(n1, t)D1, P(n, 0) 0, for n?1.
  
• Theorem 4.4 For an MAP (N(t), I(t)), t?0 with
  a matrix representation (D0, D1), define
  ,
  for z?0. Then

25
4. Construction of MAPs and BMAPs (cont.)

• Properties of the MAP (N(t), I(t)), t?0
• Based on Theorem 4.4, the average number of
  arrivals per unit time, called the arrival rate,
  can be found as ? ?D1e, where ? is the
  stationary distribution of D (assuming that D is
  irreducible), i.e., ?D 0 and ?e 1.
• Three definitions of the MAP (N(t), I(t)), t?0
• Definition 4.1 (Based on Poisson proceses)
• Lemma 4.1 (Based on sojourn time and embedded MC)
• Theorem 4.3 (Based on MC)

26
4. Construction of MAPs and BMAPs (cont.)

• Construction of BMAPs The construction is based
  on Poisson processes. Parameters
• ?i, 1 ? i ? m be nonnegative numbers with a
  unit sum (i.e., ?1?m 1) For the
  determination of the initial phase.
• d0,(i,j), 1 ? i ? j ? m, dn,(i,j), 1 ? i, j ?
  m are nonnegative numbers, for 1 ? n ? N lt ?,
  and m is a finite positive integer. Assume
  d0,(i,i) ? d0,(i,1) d0,(i,
  i1)d0,(i,i1) d0,(i,m) d1,(i,1)
  d1,(i,i) d1,(i,m) dN,(i,1)
  dN,(i,m) gt 0, for 1 ? i ? m. .

27
4. Construction of MAPs and BMAPs (cont.)

• Definition 4.2
• Define m((N1)m1) independent Poisson processes
  with parameters d0,(i,j), 1 ? i ? j ? m and
  dn,(i,j), 1 ? i, j ? m, 1 ? n ? N. If dn,(i,j)
  0, the corresponding Poisson process has no
  event.
• Determine I(0) by the probability distribution
  ?i, 1 ? i ? m. Set N(0) 0.
• If I(t) i, for 1 ? i ? m, I(t) and N(t) remain
  the same until the first event occurs in the
  (N1)m1 Poisson processes corresponding to
  d0,(i,j), 1 ? j ? m, j ? i and dn,(i,j), 1 ? j
  ? m, 1 ? n ? N. If the next event comes from
  the Poisson process corresponding to d0,(i,j),
  the variable I(t) changes from phase i to phase j
  and N(t) does not change at the epoch, for 1 ? j
  ? m, j ? i If the next event comes from the
  Poisson process corresponding to dn,(i,j), the
  phase variable I(t) transits from phase i to
  phase j and N(t) is increased by n at the epoch,
  i.e., that a batch of n arrivals is associated
  with the event, for 1 ? j ? m and 1 ? n ? N.

28
4. Construction of MAPs and BMAPs (cont.)

• A sample path of a BMAP with m 3 and N 2

29
4. Construction of MAPs and BMAPs (cont.)

• Properties of the BMAP (N(t), I(t)), t?0
• Similar to Theorems 4.2 to 4.4, it can be shown
  that I(t), t?0 and (N(t), I(t)), t?0 are
  CTMCs. That (N(t), I(t)), t?0 is a BMAP with
  matrix representation (D0, D1, , DN), where Dn
  (dn,(i,j)), for 0 ? n ? N. The infinitesimal
  generator of I(t), t?0 is D D0 DN. The
  conditional distributions of the number of
  arrivals in 0, t can be obtained from
  
• The arrival rate is ? ?(D12D2NDN)e, where
  ? is the stationary distribution of D (assuming
  that D is irreducible).

30
5. Construction of MMAPs

• Construction of MMAPs The construction is based
  on Poisson processes. Parameters
• Let C0 be a finite set of indices. Examples of
  C0 are i) C0 man, woman, man, woman,
  child ii) C0 1, 2, 11, 12, 21, 22, 122,
  212 iii) C0 1, 11, 111, , 11, and iv) C0
  failure, repair.
• ?i, 1 ? i ? m be nonnegative numbers with a
  unit sum (i.e., ?1?m 1) For the
  determination of the initial phase.
• d0,(i,j), 1 ? i ? j ? m, dh,(i,j), 1 ? i, j ?
  m, h ? C0 are nonnegative numbers, and m is a
  finite positive integer. Assume
• for 1 ? i ? m.

31
5. Construction of MMAPs (continued)

• Definition 5.1 We define a stochastic process
  (Nh(t), h ? C0, I(t)), t?0 as follows.
• Define independent Poisson processes with
  parameters d0,(i,j), 1 ? i ? j ? m and
  dh,(i,j), 1 ? i, j ? m, h ? C0. If d0,(i,j)
  0 or dh,(i,j) 0, the corresponding Poisson
  process has no event.
• Determine I(0) by the probability distribution
  ?i, 1 ? i ? m. Set Nh(0) 0, for h ? C0.
• If I(t) i, for 1 ? i ? m, I(t) and Nh(t), h ?
  C0 remain the same until the first event occurs
  in the Poisson processes corresponding to
  d0,(i,j), 1 ? j ? m, j ? i and dh,(i,j), 1 ? j
  ? m, h ? C0. If the next event comes from the
  Poisson process corresponding to d0,(i,j), the
  variable I(t) changes from phase i to phase j and
  Nh(t), h ? C0 do not change at the epoch, for 1
  ? j ? m, j ? i If the next event comes from the
  Poisson process corresponding to dh,(i,j), the
  phase variable I(t) changes from phase i to phase
  j, Nh(t) is increased by one (or by a
  pre-specified number, such as the batch size) at
  the epoch, and Nl(t) remains the same for l?h and
  l?C0, for 1 ? i, j ? m, h ? C0.

32
5. Construction of MMAPs (continued)

• Properties of MMAPs
• Similar to Theorems 4.2, 4.3, and 4.4, it can be
  shown that
• I(t), t?0 and (Nh(t), h ? C0, I(t)), t?0 are
  CTMCs.
• (Nh(t), h ? C0, I(t)), t?0 is called an MMAP
  with a matrix representation (D0, Dh, h ? C0),
  where Dh (dh,(i,j)), for h ? C0.
• The infinitesimal generator of I(t), t?0 is
  .
• The conditional distributions of the numbers of
  arrivals in 0, t can be obtained from
  
  .
• The arrival rate of type h arrivals is
  , where ? is the stationary distribution of D
  (Assuming that D is irreducible).

33
5. Construction of MMAPs (continued)

• Examples of MMAPs with special features
• Cyclic arrivals
• Type 1 and type 2 customers arrive cyclically.

34
5. Construction of MMAPs (continued)

• Examples of MMAPs with special features
• Bursty vs smooth
• Type 1 process is bursty, while type 2 is
  smooth.

35
5. Construction of MMAPs (continued)

• Examples of MMAPs with special features
• 3. Individual vs group
• Every type 2 arrival is accompanied by a type 1
  arrival.
• Type 2 always follows type 1
• Orders in individual batches

36
6. An Example

• Example Consider a reliability system with two
  identical units and a repairman. If both units
  are functioning, then one is in work and the
  other one is on cold standby. If the unit in
  work fails, it is sent to the repairman for
  repair and the standby unit is put in work. If
  repair is completed before failure, the repaired
  unit is on cold standby status. If failure
  occurs before repair completion, the failed unit
  has to wait for repair. A repaired unit is put
  in work immediately if the other unit has failed.
  The times to failure and repair times are
  exponentially distributed with parameters ? and
  ?, respectively.

37
6. An Example (continued)

• System state The state of each component can be
• 0 in repair,
• 1 waiting for repair,
• 2 on cold standby and
• 3 in work.
• The system has three phases (states)
• (3, 2), (3, 0), (1, 0),
• since the two units are identical.

38
6. An Example (continued)

• System state CTMC
• Let I(t) be the status of the system at time t.
  The process I(t), t?0 can be defined by the
  following underlying Poisson processes.
• For phase (3, 2) A Poisson process (with
  parameter) ?. If an event occurs, the process
  I(t), t?0 transits to phase (3, 0).
• For phase (3, 0) A Poisson process ? and a
  Poisson process ?. If an event from Poisson
  process ? occurs first, the process transits to
  phase (1, 0). If an event from Poisson process ?
  occurs first, the process transits to phase (3,
  2).
• For phase (1, 0) A Poisson process ?. If an
  event occurs, the process transits to phase (3,
  0).

39
6. An Example (continued)

• System state CTMC
• Then the infinitesimal generator of I(t), t?0
  is
• If ? 0.01 and ? 0.5, the stationary
  distribution of the CTMC I(t), t?0 is ?
  (0.9800, 0.0196, 0.0004), which, by definition,
  satisfies ?Q 0 and ?e 1, where e is the
  column vector of ones.

40
6. An Example (continued)

• Number of repairs MAP
• Let N(t) be the number of repairs in 0, t.
• (N(t), I(t)), t?0 is an MAP with a matrix
  representation
• If ? 0.01 and ? 0.5, the stationary
  distribution of I(t), t?0 is ? (0.9800,
  0.0196, 0.0004) and the arrival rate ?D1e
  0.009996, i.e., repair completion occurs 0.009996
  times per unit time.

41
6. An Example (continued)

• Numbers of repairs and failures MMAP
• Let Nr(t) be the number of repairs completed in
  0, t and Nf(t) the number of failures occurred
  in 0, t.
• (Nr(t), Nf(t), I(t)), t?0 is an MMAP with a
  matrix representation
• If ? 0.01 and ? 0.5, the stationary
  distribution of I(t), t?0 is ? (0.9800,
  0.0196, 0.0004), the arrival rate of failures is
  ?Dfe 0.009996, and the arrival rate of repairs
  is ?Dre 0.009996.

42
6. An Example (continued)

• An extension
• Times to failure have a common PH-distribution
  with matrix representation (?, T) (see Neuts
  (1981))
• Repair times have a common PH-distribution with
  matrix representation (?, S).
• (Nr(t), Nf(t), I(t)), t?0 is an MMAP with a
  matrix representation

43
Thank you very much! Any Question?