Markovian

Markovian refers to systems or models that obey the Markov property: the conditional probability distribution of future states depends only on the present state, not on the sequence of events that preceded it. In other words, given the current state, the past is irrelevant for predicting the future.

In discrete time, a Markov chain consists of a finite or countable state space, a transition matrix where the entry P(i, j) gives the probability of moving from state i to state j in one step, and an initial distribution. The process is memoryless in the sense that future evolution depends solely on the current state. In continuous time, a Markov process uses a generator or rate matrix and a transition function that describes the distribution after time t.

Key properties include the Chapman–Kolmogorov equations, and, for irreducible chains, the existence of stationary distributions, which satisfy certain balance conditions (for discrete time, pi P = pi; for continuous time, pi Q = 0). Markovian models can have various state spaces and may be time-homogeneous (transition rules do not depend on the clock) or time-inhomogeneous.

Not all stochastic processes are Markovian; some exhibit memory that requires additional history to predict the future. Markovian models are widely used because of their mathematical tractability and wide applicability.

Examples include the M/M/1 queue, simple random walks, and many stochastic models in finance and physics. In machine learning and artificial intelligence, Markovian assumptions underpin Markov decision processes and hidden Markov models, where the hidden state evolves in a Markovian fashion and observations depend on that state.

See also: Markov chain, Markov process, Markov decision process.