ML//neural network//feature

A feature is a direction in activation space that represents a concept, such as *Michael Jordan*, *basketball*, *plural* or *code syntax*. The model learns these directions by itself; in applied machine learning the word usually means an input the engineer computes before training, and the version built from a known physical law is a physics-based feature.

A feature is a direction in activation space that represents a concept, such as Michael Jordan, basketball, plural or code syntax. The model learns these directions by itself; in applied machine learning the word usually means an input the engineer computes before training, and the version built from a known physical law is a physics-based feature.

MLP layers detect and add features: if the current embedding aligns with a feature direction above threshold (ReLU), that feature gets added. If a vector already points toward Michael and Jordan, for instance, the MLP adds the basketball direction.

Features can be distributed across neurons (superposition) or clean (monosemantic).

Sparse autoencoders try to extract the clean feature set from superposed activations.

Features are directions, not neurons. A single neuron might participate in many features.