mathematics//statistics//covariance matrix
How one matrix describes variability in every direction. Let \(X\in\mathbb R^p\) be a random vector of measured variables with mean \(\mu\). The **covariance matrix** is
How one matrix describes variability in every direction. Let X∈RpX\in\mathbb R^pX∈Rp be a random vector of measured variables with mean μ\muμ. The covariance matrix is
Σ=E[(X−μ)(X−μ)T],Σij=Cov(Xi,Xj).\Sigma=\mathbb E[(X-\mu)(X-\mu)^{\mathsf T}],\qquad \Sigma_{ij}=\operatorname{Cov}(X_i,X_j).Σ=E[(X−μ)(X−μ)T],Σij=Cov(Xi,Xj).
The diagonal holds variances, the off-diagonal covariances. It is symmetric and positive semidefinite even though some entries may be negative.
The transpose closes a scalar measure. For a combination Y=aT(X−μ)Y=a^{\mathsf T}(X-\mu)Y=aT(X−μ), Var(Y)=aTΣa\operatorname{Var}(Y)=a^{\mathsf T}\Sigma aVar(Y)=aTΣa. Σa\Sigma aΣa is a vector; aTΣaa^{\mathsf T}\Sigma aaTΣa a scalar. It is not "multiply by the transpose" and nothing more: the first vector sets the combination and the second closes the quadratic form. If Σvi=λivi\Sigma v_i=\lambda_i v_iΣvi=λivi with ∥vi∥2=1|v_i|_2=1∥vi∥2=1, then Var(viTX)=λi\operatorname{Var}(v_i^{\mathsf T}X)=\lambda_iVar(viTX)=λi: that is the statistical reading of the eigenvalues (modes).
A small example.
Σ=[5445].\Sigma=\begin{bmatrix}5&4\\4&5\end{bmatrix}.Σ=[5445].
The common mode v+=(1,1)T/2v_+=(1,1)^{\mathsf T}/\sqrt2v+=(1,1)T/2 has variance 9; the difference mode v−=(1,−1)T/2v_-=(1,-1)^{\mathsf T}/\sqrt2v−=(1,−1)T/2 has variance 1. The variables fluctuate mostly together and their difference varies less. Nothing says the difference decays exponentially: this is not an evolution matrix.
The real bridge to the Laplacian. Σ\SigmaΣ and LLL share the algebraic structure of a symmetric positive semidefinite matrix. Their quadratic forms answer different questions: aTΣaa^{\mathsf T}\Sigma aaTΣa is the variance of a combination, xTLxx^{\mathsf T}LxxTLx the disagreement over edges (graph Laplacian). A covariance is not automatically a Laplacian: its rows need not sum to zero and its cross entries do not carry the standard Laplacian signs. Turning it into an affinity requires another rule.
Units and preparation. Σij\Sigma_{ij}Σij has units [Xi][Xj][X_i][X_j][Xi][Xj]. Mixing pressures and temperatures makes the Euclidean geometry depend on the choice of units. Centering subtracts means; standardizing divides by standard deviations. Different operations, and they change which variations PCA favors. The worked case is redundant sensors.
Reference for the directional-variance identity: Shalizi, Principal Components Analysis, section 16.1