ML//inference//prefill

Prefill is the first phase of inference in a language model, the pass that reads the whole prompt before the first token of the answer is generated, and it decides how long a user waits before anything appears (the time to first token tracked by every inference pool). It takes the prompt as token IDs and returns the probability distribution of the first new token and, as a side product, the KV cache of every prompt token at every attention layer, which the generation that follows reads instead of recomputing.

Prefill is the first phase of inference in a language model, the pass that reads the whole prompt before the first token of the answer is generated, and it decides how long a user waits before anything appears (the time to first token tracked by every inference pool). It takes the prompt as token IDs and returns the probability distribution of the first new token and, as a side product, the KV cache of every prompt token at every attention layer, which the generation that follows reads instead of recomputing.

The prompt is known in full, so all its tokens go through the network together, in chunks (512 tokens by default in llama.cpp), with causal masking keeping each token blind to those after it. Within a chunk every weight read from memory serves hundreds of tokens, which makes prefill compute-bound where the decoding steps that follow are memory-bound. On a laptop with a Radeon 860M integrated GPU, Qwen3.6 35B-A3B reads a prompt at about 93 tokens a second and writes at about 12.

Prefill time grows with the length of the prompt and shrinks with the processor's arithmetic, not with its memory bandwidth. A request that is slow to start is a prefill problem; one that writes slowly is a decoding problem, and the two have different fixes.

The cost is about two operations per active parameter per prompt token. A 6,500-token prompt through a model with 3 billion active parameters is about 39 trillion operations; at the 0.6 to 0.75 trillion useful operations a second of that laptop GPU, a little over a minute.

Each chunk costs at least one read of the weights it touches, so a very short prompt costs about as much as one decoding step. Past that floor the time rises chunk by chunk, close to a straight line, and the attention scores add a term that grows with the square of the length and matters only in long contexts.

The levers sit on the prompt side: fewer and shorter retrieved passages, a stable beginning that prefix caching can skip, a cache saved to disk and restored, or more arithmetic (a discrete GPU reads the same prompt tens of times faster). Ollama reports prefill as its prompt eval rate.