mathematics//statistics//covariance matrix
The covariance matrix is the symmetric matrix that collects the variances and covariances of a set of measured variables, describing in one object how they vary in every direction, so that the variance of any combination of them can be read from it. Let \(X\in\mathbb R^p\) be a random vector of measured variables with mean \(\mu\). The **covariance matrix** is
The covariance matrix is the symmetric matrix that collects the variances and covariances of a set of measured variables, describing in one object how they vary in every direction, so that the variance of any combination of them can be read from it. Let X∈RpX\in\mathbb R^pX∈Rp be a random vector of measured variables with mean μ\muμ. The covariance matrix is
Σ=E[(X−μ)(X−μ)T],Σij=Cov(Xi,Xj).\Sigma=\mathbb E[(X-\mu)(X-\mu)^{\mathsf T}],\qquad \Sigma_{ij}=\operatorname{Cov}(X_i,X_j).Σ=E[(X−μ)(X−μ)T],Σij=Cov(Xi,Xj).
The diagonal holds variances, the off-diagonal covariances. It is symmetric and positive semidefinite even though some entries may be negative, so its eigenvectors are perpendicular and its eigenvalues real and non-negative: the natural axes of the data and the variance along each.
The diagonal says how large each doubt is, and the off-diagonal how the doubts relate.
A Kalman filter carries one as PPP, and its off-diagonal terms are what let it correct the velocity when it only measures position. The mean separates the error that is calibrated away from the error that is averaged down; the covariance is where the information for fusing sensors lives.
The transpose closes a scalar measure. For a combination Y=aT(X−μ)Y=a^{\mathsf T}(X-\mu)Y=aT(X−μ), Var(Y)=aTΣa\operatorname{Var}(Y)=a^{\mathsf T}\Sigma aVar(Y)=aTΣa. Σa\Sigma aΣa is a vector; aTΣaa^{\mathsf T}\Sigma aaTΣa a scalar. The transpose does more than a multiplication: the first vector sets the combination and the second closes the quadratic form. If Σvi=λivi\Sigma v_i=\lambda_i v_iΣvi=λivi with ∥vi∥2=1|v_i|_2=1∥vi∥2=1, then Var(viTX)=λi\operatorname{Var}(v_i^{\mathsf T}X)=\lambda_iVar(viTX)=λi: that is the statistical reading of the eigenvalues (modes).
A small example.
Σ=[5445].\Sigma=\begin{bmatrix}5&4\\4&5\end{bmatrix}.Σ=[5445].
The common mode v+=(1,1)T/2\ca{v_+}=(1,1)^{\mathsf T}/\sqrt2v+=(1,1)T/2 has variance 9; the difference mode v−=(1,−1)T/2\cb{v_-}=(1,-1)^{\mathsf T}/\sqrt2v−=(1,−1)T/2 has variance 1. The common mode in green, the difference in blue. The variables fluctuate mostly together and their difference varies less. Nothing says the difference decays exponentially: this is not an evolution matrix.
The real bridge to the Laplacian. Σ\SigmaΣ and LLL share the algebraic structure of a symmetric positive semidefinite matrix. Their quadratic forms answer different questions: aTΣaa^{\mathsf T}\Sigma aaTΣa is the variance of a combination, xTLxx^{\mathsf T}LxxTLx the disagreement over edges (graph Laplacian). A covariance is not automatically a Laplacian: its rows need not sum to zero and its cross entries do not carry the standard Laplacian signs. Turning it into an affinity requires another rule.
Units and preparation. Σij\Sigma_{ij}Σij has units [Xi][Xj][X_i][X_j][Xi][Xj]. Mixing pressures and temperatures makes the Euclidean geometry depend on the choice of units. Centering subtracts means; standardizing divides by standard deviations. Different operations, and they change which variations PCA favors. The worked case is redundant sensors.
A covariance estimated from few samples distorts its own spectrum. Even when the truth is the identity, the sample matrix inflates its large eigenvalues and shrinks its small ones, by an amount the Marchenko-Pastur law predicts from the ratio of variables to samples; shrinkage toward a simple target (Ledoit-Wolf) cleans it before inverting it in a Mahalanobis distance.
Reference for the directional-variance identity: Shalizi, Principal Components Analysis, section 16.1