Privault Nicolas. Notes on Markov Chains

Unpublished, 2017. - 532 p.Stochastic and Markovian modeling are of importance to many areas of science including physics, biology, engineering, as well as in economics, finance, and social sciences. This text is an undergraduate-level introduction to the Markovian modeling of time-dependent randomness in discrete and continuous time, mostly on discrete state spaces. The emphasis is put on the understanding of concepts by examples and elementary derivations, accompanied by 72 exercises and longer problems whose solutions are completely worked out and illustrated. Some more advanced topics on Poisson stochastic integrals are also covered.The book is mostly self-contained except for its main prerequisites which consist in a knowledge of basic probabilistic concepts such as random variables, discrete distributions (essentially binomial, geometric, and Poisson), continuous distributions (Gaussian and gamma) and densities, expectation, independence, and conditional probabilities, which are recalled in the first chapter. Such basic topics can be regarded as belonging to the field of "static" probability, i.e. probability without time dependence, as opposed to the contents of this text which is dealing with random evolution over time. Our treatment of time-dependent randomness revolves around the important technique of first step analysis which is applied to random walks, branching processes, and more generally to Markov chains in discrete and continuous time, with application to the computation of ruin probabilities and mean hitting times. In addition to the treatment of Markov chains, a brief introduction to martingales is given in discrete time. This provides a different way to recover the computation of ruin probabilities and mean hitting times which was originally presented in the Markovian framework. Spatial Poisson processes on abstract spaces are also considered without any time ordering, with the inclusion of some recent results on deviation inequalities and moment identities for Poisson stochastic integrals.