6. Markov Models

Learning objectives

  • Discuss the Markov Property
  • Introduce MCMC

Tabular State Space

fairy tale generator

image credit: Aja Hammerly

Trajectories

once, upon, a, time, a, bird, and, a, mouse

a, sausage, entered, into, a, partnership, and, set

bird, a, and, set, up, house, together

Markov Property

The future of a stochastic process is independent of its past

\[P(X_{t+1} = x|X_{t}, X_{t-1}, ..., X_{t-k}) = P(X_{t+1} = x|X_{t})\] * memoryless property

Metropolis-Hastings

Metropolis-Hastings Algorithm

Markov Chain Monte Carlo

MCMC to posterior dist