ML//unsupervised learning//PCA

Which lower-dimensional linear representation keeps the most variance of the data. **PCA**, principal component analysis, builds a linear subspace oriented along the directions of largest variability. It is a data reduction and analysis method; its foundations are the covariance matrix and linear algebra.

Which lower-dimensional linear representation keeps the most variance of the data. PCA, principal component analysis, builds a linear subspace oriented along the directions of largest variability. It is a data reduction and analysis method; its foundations are the covariance matrix and linear algebra.

Objects before the algorithm. X∈RN×pX\in\mathbb R^{N\times p}X∈RN×p holds NNN samples in rows and ppp variables in columns. Subtracting each variable's mean gives XcX_cXc​, and the sample covariance is Σ^=XcTXc/(N−1)\widehat\Sigma=X_c^{\mathsf T}X_c/(N-1)Σ=XcT​Xc​/(N−1). Its unit eigenvectors vi∈Rpv_i\in\mathbb R^pvi​∈Rp are directions of variables; its eigenvalues λ1≥⋯≥λp≥0\lambda_1\ge\cdots\ge\lambda_p\ge0λ1​≥⋯≥λp​≥0 are variances along them.

Represent and reconstruct. With Ur=[v1  ⋯  vr]U_r=[v_1;\cdots;v_r]Ur​=[v1​⋯vr​],

A=XcUr,X^=XcUrUrT+1μT.A=X_cU_r,\qquad \widehat X=X_cU_rU_r^{\mathsf T}+\mathbf1\mu^{\mathsf T}.A=Xc​Ur​,X=Xc​Ur​UrT​+1μT.

The rows of AAA are the reduced coordinates of the samples. The cumulative explained variance is ∑i≤rλi/∑i≤pλi\sum_{i\le r}\lambda_i/\sum_{i\le p}\lambda_i∑i≤r​λi​/∑i≤p​λi​ when the total variance is nonzero. In the population formulation the expected squared reconstruction error equals the sum of the discarded eigenvalues. This optimality concerns orthogonal linear projections and that metric, not any engineering objective (projection).

What the user of the method decides. Centering and standardizing are not the same. If a variable changes from meters to millimeters, PCA on the covariance can favor it by scale; standardizing gives a correlation-based geometry but can amplify low-variability variables dominated by noise. The number of components depends on purpose: keeping 90 % of the variance does not mean keeping 90 % of the diagnostic, causal or predictive information (variance). The worked case is redundant sensors.

What must not be read as dynamics. Principal components are not automatically vibration modes or Jacobian eigenvectors of the plant. They describe the dataset and its scale. A time series can be fed to PCA, but computing a covariance does not model the time order (modes).

spectral clustering uses eigenvectors of a graph of samples for a different purpose; K-means can run on PCA coordinates, but grouping is another operation.

Reference: Shalizi, Principal Components Analysis, chapter 16 of the data-analysis notes