Markov chains Theory, Algorithms and Applications

Markov Chains


I dedicate this book especially to two exceptional people,
my father and my mother.


Markov Chains
Theory, Algorithms and Applications

Bruno Sericola

Series Editor
Nikolaos Limnios


First published 2013 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd
27-37 St George’s Road
London SW19 4EU
UK

John Wiley & Sons, Inc.
111 River Street
Hoboken, NJ 07030
USA

www.iste.co.uk

www.wiley.com

© ISTE Ltd 2013
The rights of Bruno Sericola to be identified as the author of this work have been asserted by him in
accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2013936313
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN: 978-1-84821-493-4

Printed and bound in Great Britain by CPI Group (UK) Ltd., Croydon, Surrey CR0 4YY


Table of Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

Chapter 1. Discrete-Time Markov Chains . . . . . . . . . . . . . . . . . . . .

1

1.1. Definitions and properties . . . . . . . . . . .
1.2. Strong Markov property . . . . . . . . . . . .
1.3. Recurrent and transient states . . . . . . . . .
1.4. State classification . . . . . . . . . . . . . . .
1.5. Visits to a state . . . . . . . . . . . . . . . . .
1.6. State space decomposition . . . . . . . . . . .
1.7. Irreducible and recurrent Markov chains . . .
1.8. Aperiodic Markov chains . . . . . . . . . . .
1.9. Convergence to equilibrium . . . . . . . . . .
1.10. Ergodic theorem . . . . . . . . . . . . . . . .
1.11. First passage times and number of visits . .
1.11.1. First passage time to a state . . . . . .
1.11.2. First passage time to a subset of states
1.11.3. Expected number of visits . . . . . . .
1.12. Finite Markov chains . . . . . . . . . . . . .
1.13. Absorbing Markov chains . . . . . . . . . .
1.14. Examples . . . . . . . . . . . . . . . . . . . .
1.14.1. Two-state chain . . . . . . . . . . . . .
1.14.2. Gambler’s ruin . . . . . . . . . . . . .
1.14.3. Success runs . . . . . . . . . . . . . . .
1.15. Bibliographical notes . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

1
5
8
12
14
18
22
30
34
41
53
53
58
64
68
70

76
76
78
82
87

. . . . . . . . . . . . . . . . .

89

2.1. Definitions and properties . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2. Transition functions and infinitesimal generator . . . . . . . . . . . . . .

92
93

Chapter 2. Continuous-Time Markov Chains

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.


vi

Markov Chains – Theory, Algorithms and Applications

2.3. Kolmogorov’s backward equation . . . . . .
2.4. Kolmogorov’s forward equation . . . . . . .
2.5. Existence and uniqueness of the solutions . .
2.6. Recurrent and transient states . . . . . . . . .
2.7. State classification . . . . . . . . . . . . . . .
2.8. Explosion . . . . . . . . . . . . . . . . . . . .
2.9. Irreducible and recurrent Markov chains . . .
2.10. Convergence to equilibrium . . . . . . . . .
2.11. Ergodic theorem . . . . . . . . . . . . . . . .
2.12. First passage times . . . . . . . . . . . . . .
2.12.1. First passage time to a state . . . . . .
2.12.2. First passage time to a subset of states
2.13. Absorbing Markov chains . . . . . . . . . .
2.14. Bibliographical notes . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

108
114
127
130
137
141
148
162
166
172
172
177
184
190

Chapter 3. Birth-and-Death Processes . . . . . . . . . . . . . . . . . . . . . . 191
3.1. Discrete-time birth-and-death processes . . . . . . . .
3.2. Absorbing discrete-time birth-and-death processes . .
3.2.1. Passage times and convergence to equilibrium .
3.2.2. Expected number of visits . . . . . . . . . . . . .
3.3. Periodic discrete-time birth-and-death processes . . .
3.4. Continuous-time pure birth processes . . . . . . . . .
3.5. Continuous-time birth-and-death processes . . . . . .
3.5.1. Explosion . . . . . . . . . . . . . . . . . . . . . .
3.5.2. Positive recurrence . . . . . . . . . . . . . . . . .
3.5.3. First passage time . . . . . . . . . . . . . . . . .
3.5.4. Explosive chain having an invariant probability .
3.5.5. Explosive chain without invariant probability . .
3.5.6. Positive or null recurrent embedded chain . . . .

3.6. Absorbing continuous-time birth-and-death processes
3.6.1. Passage times and convergence to equilibrium .
3.6.2. Explosion . . . . . . . . . . . . . . . . . . . . . .
3.7. Bibliographical notes . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.

191
200
201
204
208
209
213
215
217
220
225
226
227
228
229
231
233

Chapter 4. Uniformization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
4.1. Introduction . . . . . . . . . . . . . . . . . . . . .
4.2. Banach spaces and algebra . . . . . . . . . . . .
4.3. Infinite matrices and vectors . . . . . . . . . . . .
4.4. Poisson process . . . . . . . . . . . . . . . . . . .
4.4.1. Order statistics . . . . . . . . . . . . . . . .
4.4.2. Weighted Poisson distribution computation

4.4.3. Truncation threshold computation . . . . .
4.5. Uniformizable Markov chains . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

235
237
243
249
252
255
258
263


Table of Contents

4.6. First passage time to a subset of states . . . . . .
4.7. Finite Markov chains . . . . . . . . . . . . . . . .
4.8. Transient regime . . . . . . . . . . . . . . . . . .
4.8.1. State probabilities computation . . . . . . .
4.8.2. First passage time distribution computation
4.8.3. Application to birth-and-death processes .
4.9. Bibliographical notes . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.

.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.

.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.

.
.
.

vii

273
275
276
276
280
282
286

Chapter 5. Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
5.1. The M/M/1 queue . . . . . . . . . . . . . . . .
5.1.1. State probabilities . . . . . . . . . . . . .
5.1.2. Busy period distribution . . . . . . . . . .
5.2. The M/M/c queue . . . . . . . . . . . . . . . . .
5.3. The M/M/∞ queue . . . . . . . . . . . . . . . .
5.4. Phase-type distributions . . . . . . . . . . . . .
5.5. Markovian arrival processes . . . . . . . . . . .
5.5.1. Definition and transient regime . . . . . .
5.5.2. Joint distribution of the interarrival times
5.5.3. Phase-type renewal processes . . . . . . .
5.5.4. Markov modulated Poisson processes . .
5.6. Batch Markovian arrival process . . . . . . . .
5.6.1. Definition and transient regime . . . . . .
5.6.2. Joint distribution of the interarrival times
5.7. Block-structured Markov chains . . . . . . . .

5.7.1. Transient regime of SFL chains . . . . . .
5.7.2. Transient regime of SFR chains . . . . . .
5.8. Applications . . . . . . . . . . . . . . . . . . . .
5.8.1. The M/PH/1 queue . . . . . . . . . . . . .
5.8.2. The PH/M/1 queue . . . . . . . . . . . . .
5.8.3. The PH/PH/1 queue . . . . . . . . . . . .
5.8.4. The PH/PH/c queue . . . . . . . . . . . .
5.8.5. The BMAP/PH/1 queue . . . . . . . . .
5.8.6. The BMAP/PH/c queue . . . . . . . . .
5.9. Bibliographical notes . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

288
290
311
315
318
323
326
326
336
341
342
342
342
349
352

354
363
370
370
372
372
373
376
377
380

Appendix 1. Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395



Preface

Markov chains are a fundamental class of stochastic processes. They are very
important and widely used to solve problems in a large number of domains such as
operational research, computer science and distributed systems, communication
networks, biology, physics, chemistry, economics, finance and social sciences. The
success of Markov chains is mainly due to their simplicity of use, the large number of
available theoretical results and the quality of algorithms developed for the numerical
evaluation of various metrics associated with them.
The Markov property means that, for a fixed stochastic process, if the state of the
process is known at a given time then its past and future, with respect to this time,
are independent. In other words, if the state of the process is known at a given time,
predicting its future with regard to this point does not require any information about

its past. This property allows for a considerable reduction of parameters necessary to
represent the evolution of a system modeled by such a process. It is simple enough for
the modeling of systems to be natural and intuitive but also very rich in that it allows
us to take into account general probability distributions in a very precise manner.
This flexibility in modeling also allows us to consider phenomena such as
synchronization or, more generally, stochastic dependencies between components of
a system or between a system and its environment. However, this flexibility of use
may lead to either an increase in the number of states of the process in the finite case
or an increase in its structural complexity in the infinite case.
This book is devoted to the study of discrete-time and continuous-time Markov
chains on a countable state space. This study is both theoretical and practical
including applications to birth-and-death processes and queuing theory. It is
addressed to all researchers and engineers in need of stochastic models to evaluate
and predict the behavior of systems they develop. It is particularly appropriate for
academics, students, engineers and researchers in computer science, communication
networks and applied probability.


x

Markov Chains – Theory, Algorithms and Applications

It is structured as follows. Chapter 1 deals with discrete-time Markov chains. We
describe not only their stationary behavior with the study of convergence to
equilibrium and ergodic theorem but also their transient behavior together with the
study of the first passage times to a state and a subset of states. We also consider in
this chapter finite Markov chains as well as absorbing Markov chains. Finally, three
examples are proposed and covered thoroughly.
Chapter 2 discusses continuous-time Markov chains. We detail precisely the way
the backward and forward Kolmogorov equations are obtained and examine the

existence and uniqueness of their solutions. We also treat in this chapter the
phenomenon of explosion. This occurs when the Markov chain undergoes infnitely
many jumps in a finite interval of time. We show the way to obtain the main results
related to this phenomenon. As with the discrete case, we describe the stationary
behavior of these chains with the study of convergence to equilibrium and ergodic
theorem. We fnally propose an analysis of the first passage times to a state and a
subset of states as well as a study of absorbing Markov chains.
Chapter 3 is devoted to the particular case of birth-and-death processes. These
processes are characterized by a tridiagonal transition probability matrix, in the
discrete-time case, and by a tridiagonal transition rate matrix, in the continuous-time
case. In this chapter, we apply the results obtained from Chapters 1 and 2 concerning
passage times and the average number of visits. We examine the way to obtain the
explosion conditions in function of the transition rates and we show that the existence
of an invariant probability does not ensure that the chain will be non-explosive.
Finally, we give examples of positive recurrent and null recurrent continuous-time
birth-and-death processes for which the embedded chains no longer possess these
properties.
Chapter 4 deals with uniformization that, for a given continuous-time Markov
chain, consists of the construction of a stochastically equivalent chain such that the
sojourn times in each state have the same exponential distribution. This equivalent
chain connects continuous time to discrete time by the sole intermediary of the
Poisson process that we examine carefully. Nevertheless, not every Markov chain can
be uniformized. This requires that the sequence of exit rates of each state be
bounded. We are then placed inside the framework of Banach spaces and algebra
allowing the manipulation of infinite matrices and vectors. This property of
uniformization is of particular importance because it allows simple and accurate
numerical evaluation of various metrics such as state probabilities of the given
Markov chain and the distribution of the first passage times to a subset of states for
which we provide the associated computational algorithms both in the general case
and in the particular case of uniformizable birth-and-death processes.

Chapter 5 discusses the transient behavior of Markovian queues mainly for
calculating state probabilities at a given time as well as for calculating the


Preface

xi

distribution of busy periods of (a) server(s). We first consider the M/M/1 queue for
which we obtain simple formulas using generating functions. These techniques do
not directly extend to the M/M/c queue, in which case we recommend the use of the
algorithms proposed in Chapter 4 for the uniformizable birth-and-death processes.
The M/M/∞ queue does not lead to a uniformizable Markov chain but its state
probabilities at every instant are obtained in a simple manner. The distribution of the
busy periods of the servers is more diffcult to obtain. The other queues that we
propose to analyze are more general but they lead to uniformizable Markov chains.
Their sometimes complex structure generates block-structured Markov chains whose
transient behavior will be examined carefully. The treatment of these complex queues
is motivated by their use in the domain of performance evaluation of communication
networks.
Each chapter ends with bibliographic notes allowing the reader to complete or
pursue the study of certain specific aspects of his or her choice. Finally, an appendix
summarizes the basic results of integration and probability theory used throughout the
book.
There are many books on Markov chains, which generally deal with steady-state
analysis. The uniqueness of this book lies in the fact that it offers, in addition, a
detailed study of the first explosion time, backward and forward Kolmogorov
equations, birth-and-death processes as well as of uniformizable Markov chains and
the treatment of transient behavior with associated algorithms and applications to
general queues.

I would like to end this preface by thanking Nikolaos Limnios, who heads this
collection, for his proposal to carry out this work. I also thank very warmly the
reviewers François Castella, Jean-Louis Marchand and Coralie Sericola for their
valuable work and the great relevance of their numerous comments and suggestions.
Last but not least, my thoughts go to my wife, my two children, my brother and all
my family who supported me all along this work.



Chapter 1

Discrete-Time Markov Chains

We consider in this chapter a collection of random variables X = {Xn , n ∈
} defined on a probability space (Ω, F, ), with values in a countable set S and
satisfying the Markov property, that is the past and the future of X are independent
when its present state is known. Time is represented here by the subscript n, which is
the reason we refer to discrete time. The set S is often called the state space.
1.1. Definitions and properties
D EFINITION 1.1.– A stochastic process X = {Xn , n ∈
discrete-time Markov chain if:

} on a state space S is a

– for all n ≥ 0, Xn ∈ S,
– for all n ≥ 1 and for all i0 , . . . , in−1 , in ∈ S, we have:
{Xn = in | Xn−1 = in−1 , . . . , X0 = i0 } =

{Xn = in | Xn−1 = in−1 }.

D EFINITION 1.2.– A discrete-time Markov chain X = {Xn , n ∈ } on a state
space S is said to be homogeneous if, for all n, k ∈ and, for all i, j ∈ S, we have:
{Xn+k = j | Xk = i} =

{Xn = j | X0 = i}.

All the following Markov chains are considered homogeneous. The term Markov
chain in this chapter will thus designate a homogeneous discrete-time Markov chain.


2

Markov Chains – Theory, Algorithms and Applications

We consider, for all i, j ∈ S, Pi,j = {Xn = j | Xn−1 = i} and we define the
transition probability matrix P of the Markov chain X as:
P = (Pi,j )i,j∈S .
We then have, by definition, for all i, j ∈ S,
Pi,j ≥ 0 and

Pi,j = 1.
j∈S

A matrix for which these two properties hold is called a stochastic matrix. For
all n ∈ , we write (P n )i,j the coefficient (i, j) of the matrix P n , where we define
P 0 = I, with I the identity matrix whose dimension will be contextually given further
on – here it is equal to the number of states |S| of S. We write α = (αi , i ∈ S) the
row vector containing the initial distribution of the Markov chain X, defined by:
αi =

{X0 = i}.

For all i ∈ S, we thus have:
αi ≥ 0 and

αi = 1.
i∈S

T HEOREM 1.1.– The process X = {Xn , n ∈ } on the state space S is a Markov
chain with initial distribution α and transition probability matrix P if and only if for
all n ≥ 1 and for all i0 , . . . , in ∈ S, we have:
{Xn = in , . . . , X0 = i0 } = αi0 Pi0 ,i1 · · · Pin−1 ,in .
P ROOF.– If X = {Xn , n ∈

[1.1]

} is a Markov chain then:

{Xn = in , . . . , X0 = i0 }
=

{Xn = in | Xn−1 = in−1 } {Xn−1 = in−1 , . . . , X0 = i0 }

=

{Xn−1 = in−1 , . . . , X0 = i0 }Pin−1 ,in .

Iterating this calculation n − 1 times over
obtain:
{Xn = in , . . . , X0 = i0 } =

{Xn−1 = in−1 , . . . , X0 = i0 }, we

{X0 = i0 }Pi0 ,i1 · · · Pin−1 ,in

= αi0 Pi0 ,i1 · · · Pin−1 ,in .


Discrete-Time Markov Chains

Conversely, if relation [1.1] is satisfied then we have
n ≥ 1,

3

{X0 = i0 } = αi0 and, for

{Xn = in | Xn−1 = in−1 , . . . , X0 = i0 }
=
=

{Xn = in , Xn−1 = in−1 , . . . , X0 = i0 }
{Xn−1 = in−1 , . . . , X0 = i0 }
αi0 Pi0 ,i1 · · · Pin−1 ,in
αi0 Pi0 ,i1 · · · Pin−2 ,in−1

= Pin−1 ,in
=

{Xn = in | Xn−1 = in−1 }.

X is, therefore, a Markov chain with initial distribution α and transition probability
matrix P .
This result shows that a discrete-time Markov chain is completely determined by
its initial distribution α and transition probability matrix P .
In the following sections, we will often use products of infinite matrices or vectormatrix or matrix-vector products of infinite dimension. Remember that, in general,
these products are not associative except when the affected matrices or vectors have
non-negative coefficients, which is the case in this chapter. More details on this subject
are given in the first chapter of [KEM 66] and in section 4.3.
T HEOREM 1.2.– If X is a Markov chain on the state space S, with initial distribution
α and transition probability matrix P then, for all i, j ∈ S and, for all n ≥ 0, we have:
1)

{Xn = j | X0 = i} = (P n )i,j ;

2)

{Xn = j} = (αP n )j .

P ROOF.–
1) For all m, n ≥ 0, we have:
{Xn+m = j | X0 = i}
=

{Xn+m = j, Xn = k | X0 = i}
k∈S

=

{Xn+m = j | Xn = k, X0 = i} {Xn = k | X0 = i}

k∈S

=

{Xn+m = j | Xn = k} {Xn = k | X0 = i}
k∈S

=

{Xm = j | X0 = k} {Xn = k | X0 = i},
k∈S


4

Markov Chains – Theory, Algorithms and Applications

where the third equality uses the Markov property and the fourth uses the homogeneity
of X.
Defining Pi,j (n) =

{Xn = j | X0 = i}, this last relation becomes:

Pi,j (n + m) =

Pi,k (n)Pk,j (m),
k∈S

that is if P (n) denotes the matrix with coefficients Pi,j (n),
P (n + m) = P (n)P (m).

These equations are called the Chapman–Kolmogorov equations. In particular, as
P (1) = P , we have:
P (n) = P (n − 1)P = P (n − 2)P 2 = · · · = P n .
2) We obtain, using point 1,
{Xn = j} =

{Xn = j | X0 = i} {X0 = i}
i∈S

αi (P n )i,j

=
i∈S

= (αP n )j ,
which completes the proof.
In particular, this result shows that if P is stochastic then P n is also stochastic, for
all n ≥ 2.
T HEOREM 1.3.– If X is a Markov chain then, for all n ≥ 0, 0 ≤ k ≤ n, m ≥ 1, for
all ik , . . . , in ∈ S and j1 , . . . , jm ∈ S, we have:
{Xn+m = jm , . . . , Xn+1 = j1 | Xn = in , . . . , Xk = ik }
=

{Xm = jm , . . . , X1 = j1 | X0 = in }.


Discrete-Time Markov Chains

5

P ROOF.– Using theorem 1.1, we have:
{Xn+m = jm , . . . , Xn+1 = j1 | Xn = in , . . . , Xk = ik }
=

{Xn+m = jm , . . . , Xn+1 = j1 , Xn = in , . . . , Xk = ik }
{Xn = in , . . . , Xk = ik }
{Xn+m = jm , . . . , Xn+1 = j1 , Xn = in , . . . , X0 = i0 }

=

i0 ,...,ik−1 ∈S

{Xn = in , . . . , X0 = i0 }
i0 ,...,ik−1 ∈S

αi0 Pi0 ,i1 · · · Pik−1 ,ik Pik ,ik+1 · · · Pin−1 ,in Pin ,j1 · · · Pjm−1 ,jm
=

i0 ,...,ik−1 ∈S

αi0 Pi0 ,i1 · · · Pik−1 ,ik Pik ,ik+1 · · · Pin−1 ,in
i0 ,...,ik−1 ∈S

= Pin ,j1 · · · Pjm−1 ,jm
=

{Xm = jm , . . . , X1 = j1 | X0 = in },

which completes the proof.
The Markov property seen so far stated that the past and the future are

independent when the present is known at a given deterministic time n. The strong
Markov property allows us to extend this independence when the present is known at
a particular random time which is called a stopping time.
1.2. Strong Markov property
Let X = {Xn , n ∈ } be a Markov chain on the state space S, defined on the
probability space (Ω, F, ). For all n ≥ 0, we denote by Fn the σ-algebra of events
expressed as a function of X0 , . . . , Xn , that is:
Fn = {ω ∈ Ω | (X0 (ω), . . . , Xn (ω)) ∈ Bn }, Bn ∈ P(S n+1 ) ,
where, for a set E, P(E) denotes the set of all subsets of E and S n+1 is the set of all
(n + 1)-dimensional vectors, whose entries are states of S. For all i ∈ S, we write
δ i = (δji , j ∈ S) the probability distribution concentrated on the state i, defined by:
δji = 1{i=j} .
T HEOREM 1.4.– If X = {Xn , n ∈ } is a Markov chain on the state space S then,
for all n ≥ 0 and for all i ∈ S, conditional on {Xn = i}, the process {Xn+p , p ∈ }


6

Markov Chains – Theory, Algorithms and Applications

is a Markov chain with initial distribution δ i and transition probability matrix P ,
independent of (X0 , . . . , Xn ). This means that for all A ∈ Fn , for all m ≥ 1 and for
all j1 , . . . , jm ∈ S, we have:
{Xn+m = jm , . . . ,Xn+1 = j1 , A | Xn = i}
=

{Xm = jm , . . . , X1 = j1 | X0 = i} {A | Xn = i}.

P ROOF.– It is sufficient to prove the result when:
A = {Xn = in , Xn−1 = in−1 , . . . , X0 = i0 }.

Indeed, A is a countable union of disjoint events of this form, therefore, the general
case can be deduced using the σ-additivity property. It is also sufficient to consider the
case where in = i as in the contrary case the two sides are null.
Let A = {Xn = i, Xn−1 = in−1 , . . . , X0 = i0 }. We have, by the Markov
property and applying theorem 1.3,
{Xn+m = jm , . . . , Xn+1 = j1 , A | Xn = i}
=

{Xn+m = jm , . . . , Xn+1 = j1 | Xn = i, A} {A | Xn = i}

=

{Xn+m = jm , . . . , Xn+1 = j1 | Xn = i} {A | Xn = i}

=

{Xm = jm , . . . , X1 = j1 | X0 = i} {A | Xn = i},

which completes the proof.
D EFINITION 1.3.– A random variable T with values in
time for the process X if for all n ≥ 0, {T = n} ∈ Fn .

∪ {∞} is called a stopping

In the following section, we often use the variable τ (j) that counts the number of
transitions necessary to reach state j, defined by:
τ (j) = inf{n ≥ 1 | Xn = j},
where τ (j) = ∞ if this set is empty. For all j ∈ S, τ (j) is a stopping time since
{τ (j) = 0} = ∅ ∈ F0 , {τ (j) = 1} = {X1 = j} ∈ F1 and, for n ≥ 2,
{τ (j) = n} = {Xn = j, Xk = j, 1 ≤ k ≤ n − 1} ∈ Fn .

Let T be a stopping time, and FT the σ-algebra of events expressed as a function
of X0 , . . . , XT , that is:
FT = B ∈ F | ∀n ∈

, B ∩ {T = n} ∈ Fn .


Discrete-Time Markov Chains

7

T HEOREM 1.5.– S TRONG M ARKOV PROPERTY.– If X = {Xn , n ∈ } is a Markov
chain and T a stopping time for X then, for all i ∈ S, conditional on {T < ∞} ∩
{XT = i}, the process {XT +n , n ∈ } is a Markov chain with initial distribution
δ i and transition probability matrix P , independent of (X0 , . . . , XT ). This means that
for all A ∈ FT , for all m ≥ 1 and for all j1 , . . . , jm ∈ S, we have:
{XT +m = jm , . . . , XT +1 = j1 , A | T < ∞, XT = i}
=

{Xm = jm , . . . , X1 = j1 | X0 = i} {A | T < ∞, XT = i}.

P ROOF.– We have:
{XT +m = jm , . . . , XT +1 = j1 , A | T < ∞, XT = i}
=

{XT +m = jm , . . . , XT +1 = j1 , A, T < ∞, XT = i}
{T < ∞, XT = i}
∞

{XT +m = jm , . . . , XT +1 = j1 , A, T = n, XT = i}

=

n=0

{T < ∞, XT = i}

∞

{Xn+m = jm , . . . , Xn+1 = j1 , A, T = n, Xn = i}
=

n=0

{T < ∞, XT = i}

∞

{Xn+m = jm , . . . , Xn+1 = j1 , A, T = n | Xn = i} {Xn = i}
=

n=0

{T < ∞, XT = i}

∞

{Xm = jm , . . . , X1 = j1 | X0 = i} {A, T = n, Xn = i}
=

n=0

{T < ∞, XT = i}
∞

{A, T = n, XT = i}
n=0

=

{Xm = jm , . . . , X1 = j1 | X0 = i}

=

{Xm = jm , . . . , X1 = j1 | X0 = i}

=

{Xm = jm , . . . , X1 = j1 | X0 = i} {A | T < ∞, XT = i},

{T < ∞, XT = i}

{A, T < ∞, XT = i}
{T < ∞, XT = i}

where the fifth equality is obtained using theorem 1.4 because A ∩ {T = n} ∈ Fn .


8

Markov Chains – Theory, Algorithms and Applications

1.3. Recurrent and transient states
Let us recall that the random variable τ (j) that counts the number of transitions
necessary to reach state j is defined by:
τ (j) = inf{n ≥ 1 | Xn = j},
where τ (j) = ∞ if this set is empty.
For all i, j ∈ S and, for all n ≥ 1, we define:
(n)

fi,j =

{τ (j) = n | X0 = i} =

{Xn = j, Xk = j, 1 ≤ k ≤ n − 1 | X0 = i}.
(1)

For n = 1, we have, of course, fi,j =
X0 = i} = Pi,j . Hence

(n)
fi,i

{τ (j) = 1 | X0 = i} =

{X1 = j |

is the probability, starting from i, that the first return to
(n)

state i occurs at time n and, for i = j, fi,j is the probability, starting from i, that the

first visit to state j occurs at time n.
T HEOREM 1.6.– For all i, j ∈ S and, for all n ≥ 1, we have:
(P n )i,j =

n

(k)

fi,j (P n−k )j,j ,

[1.2]

k=1

recalling that (P 0 )i,j = 1{i=j} .
P ROOF.– For i, j ∈ S and n ≥ 1, we have Xn = j =⇒ τ (j) ≤ n, by definition of
τ (j). From this we obtain:
(P n )i,j =
=

{Xn = j | X0 = i}
{Xn = j, τ (j) ≤ n | X0 = i}
n

=

{Xn = j, τ (j) = k | X0 = i}
k=1
n

=

{Xn = j | τ (j) = k, X0 = i} {τ (j) = k | X0 = i}
k=1
n

=

(k)

{Xn = j | Xk = j, τ (j) = k, X0 = i}

(k)

{Xn = j | Xk = j}

fi,j
k=1
n

=

fi,j
k=1
n

=
k=1

(k)

fi,j (P n−k )j,j ,


Discrete-Time Markov Chains

9

where the fifth equality comes from the fact that {τ (j) = k} = {Xk = j, τ (j) = k}
and the penultimate equality uses the Markov property since τ (j) is a stopping time.

For all i, j ∈ S, we define fi,j as:
∞

fi,j =

{τ (j) < ∞ | X0 = i} =
n=1

(n)

fi,j .

The quantity fi,i is the probability, starting from i, that the first return to state i
occurs in a finite time and, for i = j, fi,j is the probability, starting from i, that the
first visit to state j occurs in a finite time.
(n)

The calculation of fi,j and fi,j can be carried out using the following result.
T HEOREM 1.7.– For all i, j ∈ S and, for all n ≥ 1, we have:

⎧
Pi,j
if n = 1
⎪
⎪
⎨
(n)
fi,j =
(n−1)
⎪
Pi, f ,j
if n ≥ 2
⎪
⎩
∈S\{j}

and
fi,j = Pi,j +

Pi, f ,j .
∈S\{j}
(n)

(1)

P ROOF.– From the definition of fi,j , we have, for n = 1, fi,j = Pi,j . For n ≥ 2, we
have:
(n)

{Xn = j, Xk = j, 1 ≤ k ≤ n − 1 | X0 = i}

fi,j =
=

{Xn = j, Xk = j, 2 ≤ k ≤ n − 1, X1 = | X0 = i}
∈S\{j}

=

{X1 = | X0 = i}
∈S\{j}

×

{Xn = j, Xk = j, 2 ≤ k ≤ n − 1 | X1 = , X0 = i}

=

Pi,
∈S\{j}

{Xn = j, Xk = j, 2 ≤ k ≤ n − 1 | X1 = , X0 = i}.


10

Markov Chains – Theory, Algorithms and Applications

Successively using the Markov property and the homogeneity of the Markov chain,
we obtain:

(n)

fi,j =

Pi,

{Xn = j, Xk = j, 2 ≤ k ≤ n − 1 | X1 = }

Pi,

{Xn−1 = j, Xk = j, 1 ≤ k ≤ n − 2 | X0 = }

Pi, f

(n−1)
.
,j

∈S\{j}

=
∈S\{j}

=
∈S\{j}

Summing over n, we obtain, using Fubini’s theorem, the second relation.
D EFINITION 1.4.– A state i ∈ S is called recurrent if fi,i = 1 and transient if
fi,i < 1. A Markov chain is called recurrent (respectively transient) if all its states are
recurrent (respectively transient).

D EFINITION 1.5.– A state i ∈ S is called absorbing if Pi,i = 1.
All absorbing states are recurrent. Indeed, if i is an absorbing state then, by
(n)
definition, we have fi,i = 1{n=1} and so fi,i = 1, which means that the state i is
recurrent.
T HEOREM 1.8.– The state j is recurrent if and only if:
∞

(P n )j,j = ∞.

n=1

P ROOF.– Let us resume equation [1.2] for i = j, that is:
n

n

(P )j,j =

(k)

fj,j (P n−k )j,j .

k=1

Summing over n, using Fubini’s theorem and since (P 0 )j,j = 1, we obtain:
∞

(P n )j,j =

n=1

∞

n

(k)

fj,j (P n−k )j,j

n=1 k=1
∞

=

(k)

∞

fj,j
k=1

(P n−k )j,j

n=k
∞

= fj,j
n=0

(P n )j,j = fj,j

∞

1+
n=1

(P n )j,j

.


Discrete-Time Markov Chains
∞

11

(P n )j,j < ∞ then fj,j = uj /(1 + uj ) < 1, which

It follows that if uj =
n=1

means that state j is transient.

N

(P n )j,j and assume that

Conversely, let uj (N ) =
n=1

lim uj (N ) = ∞. We

N −→∞

then have, again using equation [1.2] taken for i = j, for all N ≥ 1,
N

n

uj (N ) =
n=1 k=1
N

=

(k)

fj,j (P n−k )j,j
N

(k)

(P n−k )j,j

fj,j
k=1

n=k

N

≤

N

(k)

fj,j
k=1

(P n )j,j

n=0

≤ fj,j (1 + uj (N )) ,
and, therefore, we obtain:
fj,j ≥

uj (N )
=
1 + uj (N )

1

1+

1
uj (N )

−→ 1 when N −→ ∞,

which shows that fj,j = 1 or, in other words, that state j is recurrent.
C OROLLARY 1.1.– If state j is transient then, for all i ∈ S,
∞

(P n )i,j < ∞,

n=1

and, therefore,
lim (P n )i,j = 0 and

n−→∞

lim

n−→∞

{Xn = j} = 0.

P ROOF.– Let us resume equation [1.2], that is:
n

n

(P )i,j =
k=1

(k)

fi,j (P n−k )j,j .