ML//unsupervised learning//PCA

PCA (principal component analysis) is an unsupervised method that finds the lower-dimensional linear representation keeping the most variance of the data, by building a linear subspace oriented along the directions of largest variability. Among the techniques of dimensionality reduction it is the one whose criterion is retained variance, which is not the same as retained information. In practice it is how fifty correlated sensors on a machine are summarized by three signals, how a dataset is flattened to two axes to be looked at, how redundancy between measurements is found, and how noise living in the small directions is dropped. It is the standard first answer to the curse of dimensionality, and it only works when the variance that matters is concentrated in a few linear directions. Its foundations are the covariance matrix and linear algebra.

PCA (principal component analysis) is an unsupervised method that finds the lower-dimensional linear representation keeping the most variance of the data, by building a linear subspace oriented along the directions of largest variability. Among the techniques of dimensionality reduction it is the one whose criterion is retained variance, which is not the same as retained information. In practice it is how fifty correlated sensors on a machine are summarized by three signals, how a dataset is flattened to two axes to be looked at, how redundancy between measurements is found, and how noise living in the small directions is dropped. It is the standard first answer to the curse of dimensionality, and it only works when the variance that matters is concentrated in a few linear directions. Its foundations are the covariance matrix and linear algebra.

Σ[[5.00, 4.00], [4.00, 5.00]] λ19.00 λ21.00 explained by PC190.0 % error after projection1.00 Two sensors share a signal of variance 4.00 and carry private noise of variance 1.00. The first principal axis is their average and holds 90 % of the variance; keeping only it loses λ2 = 1.00, the disagreement between the sensors.

The figure shows two sensors reading one temperature. Project the cloud onto the first axis and watch what survives: the shared signal. What is lost is the disagreement between the sensors, which is exactly what a fault detector would want to keep (redundant sensors).

The objects come before the algorithm. X∈RN×pX\in\mathbb R^{N\times p}X∈RN×p holds NNN samples in rows and ppp variables in columns. Subtracting each variable's mean gives XcX_cXc​, and the sample covariance is Σ^=XcTXc/(N−1)\widehat\Sigma=X_c^{\mathsf T}X_c/(N-1)Σ=XcT​Xc​/(N−1). Its unit eigenvectors vi∈Rpv_i\in\mathbb R^pvi​∈Rp are directions of variables; its eigenvalues λ1≥⋯≥λp≥0\lambda_1\ge\cdots\ge\lambda_p\ge0λ1​≥⋯≥λp​≥0 are variances along them. In practice the SVD of XcX_cXc​ gives them without forming Σ^\widehat\SigmaΣ, with λi=σi2/(N−1)\lambda_i=\sigma_i^2/(N-1)λi​=σi2​/(N−1) and better conditioning.

The first rrr eigenvectors, stacked as Ur=[v1  ⋯  vr]U_r=[v_1;\cdots;v_r]Ur​=[v1​⋯vr​], represent the samples in rrr coordinates and reconstruct them from those coordinates.

A=XcUr,X^=XcUrUrT+1μT.A=X_cU_r,\qquad \widehat X=X_cU_rU_r^{\mathsf T}+\mathbf1\mu^{\mathsf T}.A=Xc​Ur​,X=Xc​Ur​UrT​+1μT.

The rows of AAA are the reduced coordinates of the samples. The cumulative explained variance is ∑i≤rλi/∑i≤pλi\sum_{i\le r}\lambda_i/\sum_{i\le p}\lambda_i∑i≤r​λi​/∑i≤p​λi​ when the total variance is nonzero. The expected squared reconstruction error equals the sum of the discarded eigenvalues. This optimality concerns orthogonal linear projections and that metric, not any engineering objective (projection).

The user of the method decides the scaling and the number of components. Variance has units: a pressure in pascals (variance around 10610^6106) beside a temperature in degrees (variance 4) makes the first component the pressure alone. Standardizing, needed when units differ, gives a correlation-based geometry but can amplify quiet variables dominated by noise. Maximum variance is not maximum importance: a critical but quiet variable can land in the discarded components, and keeping 90 % of the variance does not mean keeping 90 % of the diagnostic, causal or predictive information (variance). The worked case is redundant sensors.

Its oldest industrial use is watching a plant: fitted on weeks of normal operation, it flags samples far inside the normal plane or off it (multivariate statistical process control). Spectral clustering uses eigenvectors of a graph of samples for another purpose; K-means can run on PCA coordinates.

The reference for this note is Shalizi, Principal Components Analysis, chapter 16 of the data-analysis notes.

?Are principal components the vibration modes of the system?

No. Principal components are not automatically vibration modes or Jacobian eigenvectors of the plant. They describe the dataset and its scale. A time series can be fed to PCA, but computing a covariance does not model the time order (modes).