ML//kernel methods//k-nearest neighbors

The k-nearest neighbors method is a supervised learning method that predicts a new case by averaging the labels of the \(k\) stored examples closest to it, and it is used where *what happened to similar cases?* is the question and the answer has to be shown as cases. A compressor in an unfamiliar state is compared with the history: the five most similar operating points were three healthy runs and two that ended in a valve failure, so the estimate is a 40 % failure share, and the technician can open those five records and judge for himself.

The k-nearest neighbors method is a supervised learning method that predicts a new case by averaging the labels of the kkk stored examples closest to it, and it is used where what happened to similar cases? is the question and the answer has to be shown as cases. A compressor in an unfamiliar state is compared with the history: the five most similar operating points were three healthy runs and two that ended in a valve failure, so the estimate is a 40 % failure share, and the technician can open those five records and judge for himself.

There is no training step. The model is the dataset, kept whole and searched at every query, so the method costs nothing to fit and everything to carry: memory grows with the history, and each prediction scans it (or an index built over it). For classification the kkk neighbours vote; for regression they average; kkk trades noise for blur, with k=1k=1k=1 copying the nearest label and a large kkk washing out local structure. It is the hard version of kernel regression, where every example votes with a weight that falls smoothly with distance instead of all-or-nothing.

Nearest depends on the scales. If one feature runs from 0 to 10,000 rpm and another from 0 to 1 bar, the distance sees only the speed. Standardizing each feature (or choosing a distance with physical meaning) is part of the model, and two reasonable scalings give two different sets of neighbours.

It breaks in many dimensions. As features multiply, the distances from a query to all examples become almost equal, so the nearest is barely nearer than the farthest; this is the curse of dimensionality. The method works with a handful of well-chosen features, often after PCA or physics-based features have reduced them.

It is a poor fit for the edge. A model that must ship its whole history to a PLC or a microcontroller and search it every cycle wastes the memory and the time budget that a regression or a small tree would leave free.

Approximate search indexes make it scale to millions of vectors, the same machinery as a vector database, which is kNN over embeddings under another name.

The rest of the family is in kernel methods: the support vector machine keeps only the examples on the boundary, the Gaussian process adds an uncertainty to the average.