ML//transformer//forward pass

A forward pass is one complete pass through the network, from the input embeddings to the token it yields.

A forward pass is one complete pass through the network, from the input embeddings to the token it yields.

1Embeddings plus positional encoding2NNN layers of attention, each followed by an MLP3Layer norm4LM head5Logits6Softmaxyields the token

The residual stream carries information through all layers. Each block adds a delta through residual connections, and the final hidden state is the stream's last value.

In inference, one forward pass predicts one token (one decoding step). In training, it processes the full sequence in parallel: the causal mask ensures each position only sees prior tokens, producing T−1T-1T−1 training examples in one pass.

In generation, only the first pass reads the whole prompt (the prefill); each later pass runs one new token through the layers, starting from its plain embedding, and reads the keys and values that earlier tokens left at every layer in the KV cache. That per-token state is what separates the transformer from an RNN, which carries the whole past in one fixed-size vector.

The standard pattern is attention then MLP, repeated over NNN layers (N=96N = 96N=96 in GPT-3, N=12N = 12N=12 in GPT-2 small), each block with its residual connections and layer norm. Seen as a path through latent space, the pass alternates context (attention) and fact (MLP): one context, one fact, one context, one fact.