mathematics//statistics//estimation//maximum likelihood estimation//expectation-maximization

Expectation-maximization (EM) is an iterative algorithm for maximum likelihood estimation when part of the data is missing or hidden, which alternates between guessing the hidden part from the current parameters and refitting the parameters as if that guess were data; it is how Gaussian mixture models are fitted for clustering, how hidden Markov models are trained (the Baum-Welch algorithm), and how the noise covariances of a Kalman filter can be estimated from logged data. The hidden quantity is a **latent variable**: which operating regime produced each sample, which state a machine was in, what the true trajectory was under the noisy measurements.

Expectation-maximization (EM) is an iterative algorithm for maximum likelihood estimation when part of the data is missing or hidden, which alternates between guessing the hidden part from the current parameters and refitting the parameters as if that guess were data; it is how Gaussian mixture models are fitted for clustering, how hidden Markov models are trained (the Baum-Welch algorithm), and how the noise covariances of a Kalman filter can be estimated from logged data. The hidden quantity is a latent variable: which operating regime produced each sample, which state a machine was in, what the true trajectory was under the noisy measurements.

The two-regime case shows the loop. A pump's vibration readings come from two speeds that nobody logged, so they form two overlapping bells, and the task is the mean and spread of each. If the regime of every sample were known, the fit would be two averages; if the bells were known, each sample's regime could be inferred. EM breaks the circle by alternating. The E-step computes, with the current bells, the probability that each sample came from each regime (a reading near the boundary is, say, 60% one and 40% the other). The M-step refits each bell as a weighted average, every sample counted with its probability of belonging. Repeat until nothing changes.

Each iteration can only raise the likelihood, which makes EM stable and simple to implement, with no step size to tune. The price is that it climbs to the nearest peak: started badly it settles on a poor local maximum (two bells covering one regime), and near the top it creeps, so it is run from several starting points (often seeded by K-means) and stopped on a tolerance.

In state estimation the latent variable is the state itself. The E-step runs a Kalman filter and its backward smoother over the log (RTS smoother); the M-step re-estimates the process and measurement noise from the smoothed residuals. It is one of the principled alternatives to hand-tuning QQQ and RRR (filter tuning), valid when the model's structure is right and the log is long and rich enough.

It handles missing values in the same stroke. A sensor that dropped out for part of a test is a hidden variable like any other, and EM fills it with its conditional expectation instead of deleting the rows, which keeps the information in the other channels.

It inherits the assumptions of the model it fits. A mixture of Gaussians will find Gaussian groups in data that have none, and how many components to use is a separate choice, made with a penalized criterion or against the physics of the plant (operating regime).