ML//transformer
The transformer is a neural network architecture that stacks identical blocks of attention and a per-element MLP, so that any element of a sequence can consult any other in a single parallel step, and it is the architecture behind GPT, BERT and every modern LLM, as well as vision transformers and most speech models. Each block has two pieces, multi-head self-attention and the MLP applied to every element separately, each wrapped in a residual connection and a layer norm (a per-layer rescaling that keeps activations in range and stabilizes training), repeated over \(N\) layers (\(N = 96\) in GPT-3, \(N = 12\) in GPT-2 small).
The transformer is a neural network architecture that stacks identical blocks of attention and a per-element MLP, so that any element of a sequence can consult any other in a single parallel step, and it is the architecture behind GPT, BERT and every modern LLM, as well as vision transformers and most speech models. Each block has two pieces, multi-head self-attention and the MLP applied to every element separately, each wrapped in a residual connection and a layer norm (a per-layer rescaling that keeps activations in range and stabilizes training), repeated over NNN layers (N=96N = 96N=96 in GPT-3, N=12N = 12N=12 in GPT-2 small).
Attention treats its input as a set, so order has to be added. Every element receives a positional encoding before the first block; without it, the motor drives the pump and the pump drives the motor would be the same input. Its virtue is parallelism: all positions are processed at once on a GPU, which is what made training on enormous datasets possible. Its cost is the n×nn\times nn×n attention matrix, memory and compute growing with the square of the sequence length.
About a third of the model is attention (context routing between tokens) and about two thirds MLP (fact storage and feature detection). The path through latent space alternates context and fact: one context, one fact, one context, one fact.
Each hidden state carries DDD latent features (D=4096D = 4096D=4096 is typical), and most of them are not human-interpretable abstractions.
The context vector is the last hidden state, right before the LM Head. The last token has been thrown through so many layers that it knows exactly what it is relative to everything else, which is sufficient to predict the next one.
In inference it is a 1×D1 \times D1×D vector; in training, an ntokens×Dn_{\text{tokens}} \times Dntokens×D matrix. At inference only the last token's hidden state matters, because attention has already encoded everything into it. The attention mask computes every token's hidden state, producing T−1T-1T−1 training examples from a sequence of length TTT in one forward pass, the parallelism that training exploits. Generation, by contrast, still produces one token at a time.
Its inductive bias is the weakest of the common architectures, any element may matter, so it shines with vast data or a pretrained model and is a poor first choice for short industrial series, where persistence, ARIMA and boosting often win (time series).