mathematics//dimensionality reduction

Dimensionality reduction is the family of techniques that transform data from a high-dimensional space to a lower-dimensional one while preserving as much meaningful structure as possible, used to make data visualizable, compressible, or computationally tractable without losing the essential relationships.

Dimensionality reduction is the family of techniques that transform data from a high-dimensional space to a lower-dimensional one while preserving as much meaningful structure as possible, used to make data visualizable, compressible, or computationally tractable without losing the essential relationships.

Three techniques are common. PCA (principal component analysis) finds the directions of maximum variance and projects onto them; what a projection keeps and drops is a separate note. t-SNE preserves local neighborhood structure for 2D visualization. Random projections exploit the fact that in high dimensions, random orthogonal vectors preserve relative distances surprisingly well.

Humans do this constantly. A doctor asked how she became one is being asked to reduce a process that depended on thousands of factors to three or four key ones, and the brain performs dimensionality reduction to make that complexity communicable.

In deep learning, dimensionality reduction is how loss landscapes (which live in millions of dimensions) get visualized in 3D. The standard technique (Li et al., 2018) selects two random orthogonal directions in weight space, forms a 2D plane, and plots the loss value as the third axis. The result is a photograph of a high-dimensional surface: imperfect but informative.