ML//transformer//latent space

The latent space is the high-dimensional space of activations in which matrix multiplications route inputs into specific semantic clusters. A given context vector is a point in it, and each new token is a new step along the manifold.

The latent space is the high-dimensional space of activations in which matrix multiplications route inputs into specific semantic clusters. A given context vector is a point in it, and each new token is a new step along the manifold.

The residual stream is the trajectory through this space: each layer moves the vector to a new point, alternating between context and fact, and the last layer contains organized, routable information.

1Contextattention2FactMLP3Context4Fact

Different contexts land in different regions, which is distributional shift. Reasoning tokens before an answer position the model in a region of coherent conclusions, while direct answers land in a region of cold responses: the same model, in different neighborhoods.

The final hidden state of the last token has been refined through so many layers that it encodes exactly where it stands relative to everything else, which is enough to predict the next token.

Because logits are dot products between the context vector and the rows of the LM head, the geometry of the space decides which token is chosen; and because generation is autoregressive, the output of an LLM hardly ever leaves its path, even with low temperature (sampling adds the randomness that branches a new path from the same point).

Mixture of Experts adds dynamic activation paths that route inputs into specialized clusters through gating mechanisms: the same space, with selective computation.