mathematics//statistics//Bayesian inference//MCMC

MCMC (Markov chain Monte Carlo) is a family of sampling algorithms that draw from a posterior distribution too complicated to compute directly, by running a random walk through parameter space built to spend time in each region in proportion to its posterior probability, and it is the tool of offline Bayesian inference: wear parameters of a fleet with few observed failures, a simulator calibrated against a handful of tests. It takes a likelihood and a prior that can be evaluated at any point; it returns a long list of parameter values whose histogram is the posterior, from which means, intervals and tail probabilities are computed.

MCMC (Markov chain Monte Carlo) is a family of sampling algorithms that draw from a posterior distribution too complicated to compute directly, by running a random walk through parameter space built to spend time in each region in proportion to its posterior probability, and it is the tool of offline Bayesian inference: wear parameters of a fleet with few observed failures, a simulator calibrated against a handful of tests. It takes a likelihood and a prior that can be evaluated at any point; it returns a long list of parameter values whose histogram is the posterior, from which means, intervals and tail probabilities are computed.

The simplest version, the Metropolis algorithm, fits in one line. From the current point θ\thetaθ, propose a nearby point θ′\theta'θ′ and accept the jump with probability

α=min⁡ ⁣(1,  p(y∣θ′) p(θ′)p(y∣θ) p(θ)),\alpha=\min\!\left(1,\;\frac{p(y\mid\theta')\,p(\theta')}{p(y\mid\theta)\,p(\theta)}\right),α=min(1,p(y∣θ)p(θ)p(y∣θ′)p(θ′)​),

where p(y∣θ)p(y\mid\theta)p(y∣θ) is the likelihood and p(θ)p(\theta)p(θ) the prior. Uphill in probability the walk always moves; downhill it sometimes does, which is what lets it map the tails. The normalising constant p(y)p(y)p(y), nearly always impossible to compute, appears in both numerator and denominator and cancels: that is the whole trick.

The walk remembers only where it is, so it is a Markov chain, designed so that its stationary distribution is the posterior. After a warm-up its positions are samples of the posterior; before it, they are samples of wherever it started.

The cost is the reason it stays offline. Thousands to millions of evaluations of the model, minutes to hours on a PC with PyMC, Stan or NumPyro, and convergence diagnostics that cannot be skipped. It never runs inside a control loop; following a state in real time is the job of a Kalman filter or a particle filter.

An unconverged chain gives normal-looking numbers. The warm-up (burn-in) is discarded, several chains are started from different points and must agree (the R^\hat RR^ statistic near 1), and the samples, being autocorrelated, count for fewer independent draws than their number (effective sample size); tools such as ArviZ report both.

It is the last rung of the ladder. If the posterior is nearly Gaussian, a Laplace approximation or a maximum likelihood fit with its standard error gives almost the same answer in seconds, and for a failure rate a conjugate Beta prior gives it in a spreadsheet. MCMC pays with few data, hierarchical models (one parameter per unit, tied by a fleet distribution) or posteriors with strange shapes.