Transformers and local inference

2026-10-23

A course from zero on the transformer and on running one, built around a single request: a clinical question over a 6,000-token synthetic patient history, sent to Qwen3.6 35B-A3B on a laptop with a Radeon 860M and 64 GB of shared memory. It builds the transformer as training sees it, whole matrices in which every token meets every weight, and then shows that at inference almost all of it is frozen except one new column, with its keys and values cached and its query thrown away. The hardware half follows the weights from HBM to the tensor cores of an H100 for a dense 7B, where every step streams all 14 GB whatever the number of tokens, and derives from that the two phases of inference, the balance point near 295 tokens, batching, chunked prefill and the KV cache in bytes. Static schematics, each stating its domain, replace the earlier simulations; two interactive figures remain, and a standalone simulator follows the whole request.

Why this article26 · oct 27Read noteI have read a lot about transformers, and nearly all of it lands in one of two piles. The first is mathematical: correct, thick with indices, and it never shows the thing moving, so I would finish a paper knowing the formula for attention and with no picture of what the chip was doing. The second simplifies until the low-level usefulness is gone (the model pays attention, the cache remembers), and nothing in it lets you predict a single number. Between the two, some intuitions are simply missing, and most are about order and timing: what is computed once and what at every step, which token a vector belongs to, which phase we are in, which piece of hardware is busy while another waits. Plenty of explanations are true in the wrong time domain; the whole-matrix picture of attention that every diagram draws is exact for training and mostly wasted motion for a model that is writing. So I am redoing this article one last time, in a less serious voice than I usually allow myself, because these intuitions travel better told the way I would tell them across a table.

I bought the laptop on my desk to run models like the one in this article locally, and almost everything that follows depends on its memory, so that is where to start. It is a Lenovo ThinkPad with an AMD Ryzen AI 7 PRO 4501, and its only graphics processor is the Radeon 860M built into the same chip, with no memory of its own: the processor and the GPU share the same 64 GB of LPDDR5X (unified memory). That pool decides which models fit. Qwen3.6 35B-A3B, the mixture of experts this article follows, takes 21.7 GB at Q4_K_M; the dense Qwen3.8 27B takes 16.8 GB; a 70B such as Llama 3.1 comes to about 43 GB in the same format and fits only once the browser and its friends give memory back, because Windows and my usual apps already hold 32 GB2.

1AMD, Ryzen AI 7 PRO 450 product page: 8 cores, 16 threads, NPU up to 50 TOPS. 2Meta, Llama 3.1 70B: 70.6 billion parameters, which at the 4.89 bits per weight of Q4_K_M come to 43 GB.

What the pool does not decide is speed. It delivers about 128 GB/s, and the iGPU reads every weight out of it over that bus, at every token; an H100 reads its own HBM at 3.35 TB/s, about 26 times faster, and that ratio runs under every number in the hardware part. The model is served by Ollama. The speeds come from a day I spent measuring this laptop, the model's details from its own file, and TokenWalk, a standalone simulator I built alongside this article, follows one request from the disk to the last token.

That request is the kind a clinical copilot would get: any interaction with the new anticoagulant?, asked after a short system prompt and a synthetic patient history of about 6,000 tokens, so the patient's data never leaves the machine. On this laptop the answer takes a little over a minute to start and then comes out at about twelve words a second, for some 300 tokens, and the rest of the article explains those two numbers in the order they made sense to me.

The transformer as whole matrices

This first part is the transformer the way training sees it: every token of a sequence goes through every layer at the same time, and every weight meets every token. Most explanations move in here, buy furniture and are never heard from again.

Dot products and matrices

Everything a language model does in bulk is one operation, repeated billions of times. Take two lists of numbers of the same length, multiply them position by position, and add up the results. With x=(0.8,0.1,0.5)x = (0.8, 0.1, 0.5)x=(0.8,0.1,0.5) and w=(0.31,−1.2,0.07)w = (0.31, -1.2, 0.07)w=(0.31,−1.2,0.07) that dot product is 0.8⋅0.31+0.1⋅(−1.2)+0.5⋅0.07=0.1630.8 \cdot 0.31 + 0.1 \cdot (-1.2) + 0.5 \cdot 0.07 = 0.1630.8⋅0.31+0.1⋅(−1.2)+0.5⋅0.07=0.163. I read it as a question, how much xxx looks like www: large when they point the same way, zero when perpendicular, negative when they oppose. A neuron in a neural network is exactly this, a dot product of its inputs with its weights followed by a function that cuts off the negative part, and the weights are what training chose. A model with 35 billion parameters has 35 billion of these numbers.

Put many neurons side by side, all reading the same input, and you get a matrix: each row is one neuron's weights, and multiplying the matrix by a vector computes every row's dot product at once. A language model is a few hundred such matrices applied one after another. To multiply one vector by a matrix, every weight is fetched from memory once and used for one multiplication and one addition. Multiply sixteen vectors by the same matrix and every weight is still fetched once, but used sixteen times. Keep that sentence in a pocket (the whole hardware part of this article is that sentence wearing a GPU). For counting, a token passing through a model costs about two operations per weight it touches:

FLOPs per token≈2×Pactive\text{FLOPs per token} \approx 2 \times P_{\text{active}}FLOPs per token≈2×Pactive

Our model touches about 3 billion weights per token (the experts explain why), so about 6 billion operations.

Tokens and embeddings

A matrix wants a vector, and we have text. A tokenizer cuts the text into pieces from a fixed vocabulary and hands the network one integer per piece: common words are one piece, rare ones several, and a leading space usually belongs to the word after it. Qwen3.6's vocabulary has 248,320 entries. They are learned once, before training, by BPE: start from single bytes, glue the pair of neighbours that appears most often in a corpus into a new entry, and repeat (Qwen's file stores its 247,587 merges in order). This is one of the two figures I kept interactive, because the point is to type your own text: it trains a tokenizer on a single clinical paragraph and cuts whatever you write.

characters63 UTF-8 bytes63 words12 tokens25 vocabulary161 A byte-pair tokenizer trained on one paragraph, keeping 120 merges on top of 41 characters, cuts the sentence into 25 pieces: T | he | ·patient | ·present | s | ·with | ·f | ev | er | ·and | ·a | ·dr | y | ·co | ug | h | ·f | or | ·th | re | e | ·d | ay | s | .. As characters it would be 63 pieces, and a model writes one piece per pass through all its layers.

Letters would be simpler and cost more. English and Spanish average about four characters per token, and each written position is a full pass through the model, so a 300-token answer would take about 1,200 passes.

A chat is a list of messages and the model only continues text, so a chat template shipped in the model file flattens the list into one string, with special tokens marking who speaks. Qwen uses ChatML, and the model writes from the last line on, so the system prompt is just the first lines of a long text:

TEXT Copy

Each token ID then becomes a vector by lookup: the embedding matrix has one row per vocabulary entry, 2,048 numbers wide in Qwen3.6, and training has nudged those rows until related tokens point in related directions. Stack the vectors of an nnn-token prompt and you have a matrix XXX of n×2,048n \times 2{,}048n×2,048. The schematics and the text call each token's vector its column; in the formulas it is a row of XXX, same numbers.

Looked up one by one, dog bites man and man bites dog give the same three vectors, so something has to carry order. GPT-3 adds a learned position vector to each embedding on entry. Qwen uses RoPE, a kind of positional encoding that rotates parts of each query and key inside every attention layer by an angle proportional to the position, so comparing two of them depends on how far apart they are (its rotation base of ten million helps it reach 262,144 tokens of context).

Attention

Each token's vector now enters the residual stream, a channel that runs the whole height of the model. Every block reads a normalised copy of it and adds its result back on top (a residual connection), so a token's column is a running sum of the embedding and, in every layer, attention's contribution and then the MLP's. The word model must mean one thing in a machine learning model and another in a fashion model; attention is the block that lets machine learning write into the column of model.

An attention head has its own three learned matrices, WQhW_Q^hWQh, WKhW_K^hWKh and WVhW_V^hWVh, and multiplies every token's full vector by each, giving every token a query, a key and a value of 256 numbers in Qwen3.6. The query is what a token looks for, the key what it advertises, the value what it hands over if chosen. In the textbook picture the whole prompt goes at once: Q=XWQhQ = XW_Q^hQ=XWQh, K=XWKhK = XW_K^hK=XWKh, V=XWVhV = XW_V^hV=XWVh, each n×256n \times 256n×256. In code the heads' matrices sit side by side as one wide WQW_QWQ (in GPT-3, 12,288×12,28812{,}288 \times 12{,}28812,288×12,288 for 96 heads of 128), and the result is cut into heads after the projection, so every head sees the whole input1.

1Brown et al., Language Models are Few-Shot Learners, 2020, table 2.1: 96 layers, a residual stream of 12,288 and 96 heads of 128.

Then every query meets every key. QKTQK^TQKT is an n×nn \times nn×n matrix of scores, one row per token, and a softmax on each row turns them into weights between 0 and 1 that sum to 1. Here is one row of it, the row of a single token, with queries and keys I wrote by hand so the pronoun has someone to find.

keyq·kweight (softmax)value vthe query of its (token 7) asks every key up to it1 Amiodarone0.64 %2 raises0.13 %3 warfarin3.474 %4 levels0.64 %5 so0.23 %6 lower0.96 %7 its0.85 %8 dose−∞masked: comes later, weight 0Σ weight × v =output of its in this head: 1 × 256 One head of one layer at token 7, with toy scores. The weights add up to 1; what travels is the content of v, mostly the value of warfarin.

From its, 74% of the weight lands on warfarin, which is more focus than I have managed on anything this year. A real head learns its vectors, but the arithmetic is the same, with dkd_kdk the length of a key (dividing by its square root keeps long vectors from making the softmax all-or-nothing):

Attention(Q,K,V)=softmax⁡ ⁣(QKTdk)V\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right) VAttention(Q,K,V)=softmax(dkQKT)V

Read the last product one token at a time, as the schematic does. A token's output from a head is its row of weights times VVV, 256 numbers, each one that row of weights dotted with one column of VVV: a blend of the other tokens' values in the proportions the softmax chose. The scores only set the proportions; the content that travels is in VVV. This weighted sum is the only place in the network where one token's information reaches another. The projections, the normalisation, the output matrix and the MLP are position-wise, token by token, never looking at the others.

A layer runs several heads side by side (multi-head attention), each free to capture a different relation. Their outputs are concatenated and multiplied by one more matrix, WOW_OWO, which mixes them back to the width of the stream, and the result is added to the residual of the token that asked, and of that token only (in GPT-3, 96 heads of 128 concatenate to exactly 12,288). Qwen3.6 has 16 query heads per attention layer but only 2 sets of keys and values, each shared by 8 heads: grouped-query attention, which stores eight times fewer keys and values at a small cost in quality1.

1Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023.

The surprise in Qwen3.6's file is full_attention_interval = 4: only ten of its forty layers are this attention. The other thirty are Gated DeltaNet layers, a linear attention that keeps a fixed-size state per layer, updated by each token as in a recurrent network1. They are Qwen's other way for tokens to mix; a fixed state cannot reach back to one specific earlier token, so some full-attention layers stay.

1Yang, Kautz and Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025.

In the schematic its never looks at dose, the word after it. In a model that writes text, every score where a query meets a later key is set to minus infinity before the softmax, and e−∞=0e^{-\infty} = 0e−∞=0, so the score matrix is a triangle: the causal mask. It exists for training (one sentence of eight words gives eight prediction tasks in one pass, none of which may peek at its answer), and it turns out to be the most consequential rule in this article. The original transformer was an encoder-decoder, with an unmasked stack for the source sentence and cross-attention into it; a decoder-only model like Qwen gets the source in its prompt and reads it with masked attention, because it lies in the past.

The MLP and its experts

After attention, every column goes alone through an MLP, two or three matrix products with a nonlinearity between them. The first matrix reads as a list of patterns, each asking how much the column matches it; the nonlinearity zeroes the low scores; the second matrix adds a direction for each pattern that fired. An MLP holds tens of thousands of these pattern-direction pairs per layer, which is where most of what a model knows seems to live (and where post-training mostly leaves it, as the piece on how LLMs learn argues)1, and in a dense model the MLPs are about two thirds of the parameters.

1Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories, 2021; Meng et al., Locating and Editing Factual Associations in GPT, 2022.

A mixture of experts splits that MLP into many small ones and adds a router. Qwen3.6 has 256 experts in each of its 40 layers, each three matrices of 2,048×5122{,}048 \times 5122,048×512; the router scores all 256 for the current token, the top 8 run, and one shared expert runs for every token. The router is one more small matrix, its values set in pretraining by the same backpropagation as every other weight. Hence the name: 35 billion parameters, about 3 billion active per token. One written token wakes 8 experts per layer plus the shared one, while a prompt chunk of 512 tokens, each choosing its own eight, wakes nearly all 256. Every expert stays in memory, since the next token may pick any, so the saving is in what is read and computed per token, and it is largest when one token is in flight, which is when the model writes.

The image draws one whole layer with GPT-3's sizes, and like the other two large figures it is rendered with Manim, the open-source animation engine Grant Sanderson wrote for 3Blue1Brown1. Follow the single line up the middle, which is the stream, and the two branches that leave it and come back as sums.

1His visual walkthrough of transformers is still the best place to build intuition for attention. The smaller schematics are SVG drawn by this article's own code, and the bandwidth scatter is plotted from TokenWalk's benchmark data.

One layer on the residual stream, drawn with GPT-3's real sizes, not Qwen's. Attention: LayerNorm, 96 heads in parallel, concatenation (96 × 128 = 12,288), the output matrix, and an addition to the stream. MLP: LayerNorm, W1 (12,288 × 49,152), GELU, W2, and a second addition. Embedding and position enter only at the bottom; after 95 more layers, a last LayerNorm and the unembedding (12,288 × 50,257) feed the softmax over the vocabulary, which is not an attention softmax.

Next-token sampling

After the last layer the next word depends on one column only, the final residual of the last token. The LM head multiplies it into 248,320 scores, one per possible next token, the logits, and a softmax turns them into probabilities after every logit is divided by a temperature TTT:

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}pi=∑jezj/Tezi/T

Take seven candidates for the word after lower the warfarin, with logits I made up, from 6.1 for dose down to −1.5 for banana. At T=1T = 1T=1, dose gets 65%, INR 20% and levels 10%. At 0.5 the favourite takes 90%; at 2 it drops to 42% and banana climbs from 0.03% to almost 1%, which in a clinical copilot is roughly how you get fired. The order never changes. At the limit of zero the top token takes everything, which is greedy decoding. Top-p instead keeps the smallest set from the top whose probabilities reach ppp and renormalises it (at p=0.9p = 0.9p=0.9 and T=1T = 1T=1, dose, INR and levels, which together hold 95%), because sampling from the full distribution occasionally picks a token from the long tail that derails the rest of the text1.

1Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020.

Temperature reshapes the whole distribution and leaves the tail alive; top-p cuts the tail and leaves the survivors' ratios alone. Qwen3.8's file recommends temperature 1.0, top-p 0.95 and top-k 20 (a fixed number of candidates), and nobody knows which candidates are the top until all 248,320 dot products are done. Claude no longer accepts any of these settings: from API version 2026-10-21 a request that sets them returns an error1. The same sampler also runs inside training, since RL makes the model write its answers at a temperature before it scores them: GRPO samples 16 answers per prompt and learns only when they differ.

1Anthropic, API version 2026-10-21 migration guide, change 1, which recommends prompting to guide behaviour.

Training

Training is the loop that chooses the weights. Predict the next token at every position of a batch of text and score the predictions with one number, the loss, here cross-entropy: the negative logarithm of the probability given to the token that actually came next, so a confident wrong guess costs the most. Backpropagation works out which way every parameter should move to lower it, and each moves a small step. Thanks to the causal mask every position is a separate example inside one parallel pass; GPT-3 was trained on 300 billion tokens in batches of 3.2 million, about 94,000 steps1. Everything after pretraining, from fine-tuning with LoRA adapters to RLHF, is in the piece on how LLMs learn, which keeps this loop and changes only where the signal comes from, and shows why an RL step is half decode and half this whole-matrix pass; here it is enough that inference never changes the weights, so the model knows about your patient only what is in the request.

1Brown et al., 2020, cited above, table 2.1.

This is the transformer as every diagram draws it: whole matrices, every weight meeting every position in one pass, a full triangle of n×nn \times nn×n scores per head. For training and for reading a prompt it is exactly right. For a model writing its answer, it is almost all wasted motion.

One moving column

Change the domain: we are now in decode, the step that writes one new token of the answer, and we follow that token through all the layers. Here is the sentence from the attention section with its mask, first as training computes it (every row at once) and then as a writing model does when the eighth word arrives.

rows: the query of each word · columns: keys 1 to 8123456781 Amiodarone1002 raises77233 warfarin3327404 levels102059125 so25111430206 lower1012261721147 its437443658 dose43247416366earliersteps:frozenthis stepempty cells: −∞ before the softmax, 0 after (the mask)

One head of one layer, in the step that adds token 8. Prefill or training computes all 8 rows in one pass, one softmax per row; decode computes only the new row, and the rows above never change.

Add a word and exactly one new row appears; the rows above do not change by a single digit, because none of those words was allowed to look at the new one. This was the moment it turned for me, and it holds at every height of the model, by induction over the layers. At layer 1 an earlier token read only tokens up to itself, so its column after layer 1 can never change; layer 2 reads only those finished columns, so its column after layer 2 is finished too, and so on to the top. Every earlier token's column, at every layer, is done for good the moment it is computed. Nobody in the past ever looks forward; only the new token looks back.

Go through what that freezes. An earlier token's key and value at each layer will be read by every later token and never change, so they are stored once. Its query was used once, for its own row of weights, and is thrown away. Its scores, its softmax and its MLP pass were done when it arrived. Of all those triangles and nnn-row matrices, the only part that moves is one new column, and almost everything it goes through (normalisation, projections, output matrix, MLP) gives it the same arithmetic it would get alone. Only attention reads other tokens, and only backwards. The image follows one head through that moment in GPT-3's sizes; look at the query that is not stored, the key and value appended as new rows, and the table where the new query meets each cached key.

One attention head while generating a token, in GPT-3's real sizes, not Qwen's. The new token's normalised residual (1 × 12,288) goes through this head's query, key and value matrices, 12,288 × 128 each; its key and value become new rows of K and V, and its query is used once and not stored. Per token, the table shows the cached key, the score (query times key over the square root of 128), the softmax weight (the dots sum to 1) and the value; the weighted sum of the values is the head's output, 128 numbers. Toy numbers on a 7-token sentence.

That stored past is the KV cache. Without it, every new token would rerun the network over all 6,500 earlier tokens to rebuild their keys, about two million token-passes for a 300-token answer instead of 6,800. It holds one key and one value per token, per attention layer and per KV head, each computed from that layer's residual (a new token at layer 39 reads what the earlier tokens had become by layer 39), and never a query. In Qwen3.6 only the ten full-attention layers keep one; the DeltaNet layers keep their fixed state. The cache is, in short, the model's diary of everything it is forbidden to reconsider, which sounds healthier than most diaries. In the image the white lines are the induction; the console's memory lens shows the cache filling one column per token.

The KV cache in GPT-3's real sizes, not Qwen's: one column per token, one row per layer, a key and a value per cell. Old columns are stored and unchanged; only the new token's column is computed now. White lines: a token at layer 2 read only tokens up to itself at layer 1, so the causal mask fixes its key and value. Arcs: the new query reads the keys of its own layer. Keys and values are kept per layer and per head, from that layer's residual rather than the embedding; queries are not kept. 96 layers × 96 heads make 9,216 softmaxes per step. Below: prefill computes the whole triangular matrix at once, each row its own softmax; decode adds one row per step.

Memory grows linearly, one more entry per layer and per KV head for each token, nothing rewritten. Work grows quadratically: the ttt-th token's query reads ttt keys, so writing nnn tokens costs n(n+1)/2n(n+1)/2n(n+1)/2 dot products per head. So a decoding step is, per head, one row of QKTQK^TQKT and one weighted sum of VVV, and one column through every weight matrix of every layer. The column still meets every active weight, so every weight must come from memory for a single token, which is why writing waits on memory. And the whole-matrix picture comes back once per request, when the prompt is read. Both facts are about where the weights physically are.

The model in memory

So far the weights were numbers in formulas. Here they become bytes with an address, in a file and then in a machine's memory, and the distance from memory to the chip decides almost every number that follows.

The GGUF file and quantization

On my disk the model is one file. Ollama stores qwen3.6:35b-a3b as a few blobs under ~/.ollama/models/blobs, and the largest, 21,718,480,960 bytes, is a GGUF file, the format of llama.cpp. Forty lines of Python read its layout, and doing that once is the fastest way I know to stop treating a model as a haunted object.

GGUF file, 21.7 GBone blob in Ollama's store headerthe four bytes GGUF, version 3, 753 tensors, 36 metadata keys

metadataarchitecture, layers, heads, context, tokenizer, chat template

tensor tableeach tensor's name, shape, number format and offset

tensor datathe weights themselves, aligned, almost all of the size

The metadata holds every number used so far:

TEXT Copy

general.architecture = qwen35moe qwen35moe.block_count = 41 # 40 layers + 1 for multi-token prediction qwen35moe.embedding_length = 2048 # width of the residual stream qwen35moe.attention.head_count = 16 # query heads qwen35moe.attention.head_count_kv = 2 # key-value sets (grouped-query attention) qwen35moe.attention.key_length = 256 qwen35moe.full_attention_interval = 4 # 1 layer in 4 is full attention qwen35moe.context_length = 262144 qwen35moe.expert_count = 256 qwen35moe.expert_used_count = 8 qwen35moe.expert_feed_forward_length = 512 tokenizer.ggml.merges = [247587 entries: 'Ġ Ġ', 'ĠĠ ĠĠ', 'i n', 'Ġ t', ...]

The tensor table names every matrix with its shape and format, so reading it is reading the architecture: blk.0.attn_qkv is a Gated DeltaNet projection, blk.3.attn_q full attention, blk.0.ffn_up_exps holds 256 experts in one tensor of 2048 × 512 × 256, and the router, blk.0.ffn_gate_inp, stays in 32-bit floats.

The shapes add up to 35,505,251,456 parameters, 71 GB at the two bytes of a 16-bit float. The file is 21.7 GB because of the formats column. Quantization stores each weight in fewer bits, and the first thing I got wrong about it is that it cannot work by cutting decimals: 4 bits are sixteen possible values, full stop1. It takes a block of weights (32, say), finds one scale for the block, and stores each weight as the index of the nearest of the evenly spaced levels that scale allows; reading it back is level times scale. This is the second figure I left interactive, because the error is something you have to watch grow as the bits go down.

1Even 16 bits only approximate: np.float16(0.1) stores 0.0999755859375, the nearest number the format can hold.

stored at 4 bits (Q4_0)integers −8 to 7 + one fp16 scale bits per weight, scale included4.5 average error at 4 bitsabout 3% of the largest weight One block of 32 weights quantized: each weight is stored as a small integer and the block shares one fp16 scale, set by its largest weight; reading back is integer times scale. At 8, 4, 3 or 2 bits, with the error of each weight.

At 8 bits the error is invisible; at 4 bits, sixteen integers from −8 to 7, it averages about 3% of the block's largest weight, which is how most people run models; at 2 bits it reaches 12%. The one large weight forces a large scale and leaves the small ones few levels near zero, which the K formats soften with a second level of smaller scales. Each block also stores its scale, so bits per weight are never round: Q8_0 is 8.5. Qwen3.6's file mixes Q4_K at 4.5 bits and Q6_K at 6.56 for the matrices that hurt most when rounded, and averages 4.89, which is what Q4_K_M means in practice. A model's size is always parameters times bits per weight over eight: 35.5 billion times 4.89 over 8, 21.7 GB. The dense 7B model that GPU courses use, which I borrow below for a data-centre card, is 28 GB in fp32 (four bytes per weight), 14 GB in fp16 or bf16, 7 GB in fp8 and 3.5 GB in 4 bits.

Parameters and gigabytes are different numbers.

A slide I wrote for myself said the dense 27B model reads 17 GB per token, and I took it for an error. Both are right: 27.3 billion parameters at 4.9 bits are 16.8 GB. My notes had the converse mistake, a 20B model at 17 GB in Q4; in Q4_K_M it is about 12 GB.

Nothing in the file is executable: the engine supplies the arithmetic, the file the numbers and shapes, which is why the previous article could hard-link one file between Ollama and LM Studio.

Bandwidth and the memory ladder

Capacity decides whether a model fits and bandwidth how fast it writes, because the chip has only a few megabytes of fast memory and every weight the moving column meets crosses the bus to it, at every token. The two models I have pull these apart: the MoE holds more (21.7 GB) and reads far less per token, about 1.84 GB (its 3 billion active weights plus the parts every token passes through), while the dense 27B holds less and reads nearly all of it, about 16 GB: nine to one.

LPDDR5X-8000 makes 8,000 million transfers a second on a 128-bit bus, 16 bytes each: 128 GB/s. Thin laptops stop at 128 bits because width costs pins and power; Apple's M4 Max uses 512 bits for 546 GB/s1. A desktop card has its own VRAM on a far wider bus, 32 GB at 1,792 GB/s on an RTX 50902; a model that does not fit in it is split with the CPU, and the slow part sets the pace (ollama ps shows the split). The laptop's iGPU borrows system RAM instead, large and slow, a fair trade when a 16 GB card cannot hold this model at all. It has 8 compute units of 64 lanes each; the NPU goes unused by Ollama.

1Apple, New MacBook Pro features M4 family of chips, 30 October 2024: M4 Max with up to 546 GB/s of memory bandwidth. 2NVIDIA, Compare GeForce graphics cards: RTX 5090, 32 GB GDDR7, 1,792 GB/s; RTX 5080, 16 GB, 960 GB/s.

Each of these machines is a ladder of memories, each rung larger and slower than the one below it (memory hierarchy). The laptop's is short, disk, shared RAM and a few megabytes on the chip, and its iGPU reads the same RAM the model was loaded into. The H100 is the card whose rungs are documented1, so from here to the KV cache I follow the example every GPU course uses beside our model: a dense 7B in fp16, 14 GB, 32 layers of width 4,096, on one H100 in a server.

1NVIDIA, NVIDIA Hopper Architecture In-Depth, 2022: H100 SXM5 with 132 SMs, four tensor cores and 128 FP32 cores per SM, up to 228 KB of shared memory per SM and 50 MB of L2; the H100 product page gives 80 GB at 3.35 TB/s and 1,979 TFLOP/s of BF16 with sparsity, about 990 dense.

NVMe SSD · terabytes7 GB/s · 2 s · at loadhost RAM · 64 to 512 GBPCIe 5.0 · 64 GB/s · 220 ms · at loadHBM3 (VRAM) · 80 GB · the weights live herememory bus · 3.35 TB/s · 4.2 ms · every stepL2 cache · 50 MBon chip · never holds the model wholeSM · 228 KB × 132model loaded One server with an H100 SXM; each time is what it takes to move the whole 7B model in fp16, 14 GB. The lower rungs are faster and smaller.

A model is loaded when it sits in HBM. The two upper rungs are paid once: the SSD hands 14 GB to host RAM in about 2 s and PCIe 5.0 carries them to the card in about 220 ms, and the line between RAM and HBM is where a file becomes a model in service. HBM is the memory and the 3.35 TB/s is the bus beside it, which every weight crosses at every step: 14 GB in 4.2 ms. Leaving the weights in host RAM and pulling them over PCIe at each step (offloading) would cost 220 ms per token instead of 4.2 ms, about fifty times slower. The two lowest rungs never hold the model at all, which took me embarrassingly long to digest.

Prefill and decode on the hardware

Here is the fact the rest of the article hangs on, and the one I wish someone had told me first: the weight matrices do not care how many tokens come. The domain for this whole part is one step (one pass through all 32 layers) of the dense 7B in fp16 on one H100 SXM, at spec-sheet maxima (real systems get 60 to 80% of them); the laptop comes back at the end of each section.

The picture my Anki cards settled on is a kitchen. The weights are the ingredients, 14 GB of them, kept in the pantry (HBM). The cooks are the 132 streaming multiprocessors (SMs). Between them a kitchen hand, the memory bus, carries the ingredients over one box at a time (a tile of weights), and a step of the model is one full round: every box in the pantry carried over once, BAM BAM BAM, 14 GB at 3.35 TB/s, 4.2 ms. A token is a dish. The box costs the same to fetch whatever the number of dishes it ends up in, so with one token or with 2,048 the kitchen hand's round takes 4.2 ms, and what the number of dishes changes is how long the cooks work on each box.

Weights by the tile

Each SM has four tensor cores (units that multiply a small block of one matrix by another and accumulate, in one instruction) beside 128 ordinary lanes for the element-wise work (softmax, normalisation, activations), and a counter of 228 KB of shared memory; the 132 of them do about 990 trillion operations a second. Nothing a model is made of fits on that counter. One layer of the 7B is about 400 MB, its WQW_QWQ (4,096 × 4,096) 33 MB, one head's slice of it (4,096 × 128) 1 MB, and only a tile of 128 × 128 numbers, 32 KB, fits with room left for the input and the running sums.

So every matrix is cut into tiles (tiling): an SM puts a tile on its counter, multiplies it by the tokens it has, adds the result to its running sums and drops it, while the next tile is already on its way. Kernel libraries (cuBLAS, CUTLASS, Triton) write this; the model code only says x @ W. The model is never whole on the chip (the 132 counters add up to 30 MB, 0.2% of it). The 4.2 ms is the cumulative streaming of every weight once, a little more in practice since some tiles are read twice and activations and the KV cache travel too, and it is also the floor of decode latency: no token can be written faster than one round of the pantry.

Now the same box in both phases. In decode the dish is one token, a scrawny vector of 4,096 numbers, so each 32 KB tile serves that one token: 128 × 128 multiply-adds, about 32,000 operations. In prefill the tile meets a fat matrix of prompt tokens, and with 128 of them it yields about 4 million, for exactly the same trip.

decodeW128 × 12832 KB×1 tokenscrawny vector32K operationsprefillW128 × 12832 KB×128 tokensfat matrix4M operations One tile of one weight matrix in one SM of an H100. The tile costs the same to bring into shared memory; what changes is how many tokens use it before it is dropped.

Picture the scrawny vector standing at the end of the bus while 14 GB of weights are poured over it, 32 KB at a time, each tile used once and thrown away. It is the most expensive way ever devised to compute 4,096 numbers, and it is how every chatbot writes.

Within the step, each layer runs as eight ordered stages, each one an order to the GPU (a kernel), and inside a stage everything runs in parallel: every token, every head, every output number.

1Q, K, V (WQ, WK, WV fused)reads weights2Q · Kᵀ (scores)computes · KV*3causal mask and softmaxcomputes4weights · Vcomputes · KV*5· WO and residual addreads weights6LayerNormcomputes7MLP: · W1 and activationreads weights8MLP: · W2 and residual addreads weightsFlashAttention:one kernel* in decode, 2 and 4 also read the KV cache from HBM

One layer (pre-norm), repeated in all 32. Each stage is one kernel and the stages run in a row, with everything inside a stage at once.

The four stages that multiply by a weight matrix carry almost all the traffic; the other four only compute with what they already have (a pre-norm model such as GPT-3 or Qwen runs one more normalisation before the first stage). In decode, stages 2 and 4 also read the KV cache, the frozen past of the moving column. Between stages and between layers the order is strict, since each needs the last one's output. FlashAttention fuses stages 2 to 4 into a single kernel (kernel fusion), so the n×nn \times nn×n scores live and die in shared memory and are never written to HBM, and a running softmax keeps the result exact1.

1Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.

Step time and the balance point

The kitchen hand and the cooks work at the same time on different boxes: while the tensor cores multiply tile kkk, the bus is already bringing tile k+1k+1k+1 (double buffering; the H100 has a unit that copies asynchronously). That is overlap, a different kind of parallelism from the 132 cooks splitting one matrix between them: splitting divides the arithmetic, overlap hides the fast one's time under the slow one's.

decode, 1 token: the SMs wait for the busbusSM12345step ≈ the sum of the trips (4.2 ms)prefill, 2,048 tokens: the bus waitsbusSM12345step ≈ the sum of the arithmetic (29 ms)bus: brings tile kSMs: compute tile kwaiting

One weight matrix on an H100, tile by tile, each lane to scale. Two different units work at the same time: the bus copies, the tensor cores multiply, and the faster one hides under the slower one.

So a step lasts as long as the slower of the two, tstep=max⁡(tmemory,tcompute)t_{\text{step}} = \max(t_{\text{memory}}, t_{\text{compute}})tstep=max(tmemory,tcompute), never their sum. The kitchen hand's round is fixed at 4.2 ms; the cooks spend two operations per weight on each dish, 14 billion operations, 0.014 ms per token at 990 TFLOP/s. Scaled up, it is a kitchen hand who takes 300 seconds to bring the box and a cook who takes one second per dish. Three steps on one time axis:

1 token (decode)4.2 ms per tokenbus4.2 msSM0.014 ms · then waitsstep: 4.2 ms for 1 token295 tokens (balance)0.014 ms per tokenbus4.2 msSM4.1 msstep: 4.2 ms for 295 tokens2,048 tokens (prefill)0.014 ms per tokenbus4.2 ms · then waitsSM29 msstep: 29 ms for 2,048 tokensbus: bring weightsSMs: computewaiting

One step of a 7B in fp16 through its 32 layers on an H100, all three on one time axis. The bus brings weights while the SMs compute, and the step ends when the slower one ends.

One dish per box, a decode step: the round lasts 4.2 ms and the cooks work 0.014 ms of it, idle 99.7% of the step, for about 240 tokens a second.

About 300 dishes (295 tokens): both take 4.2 ms and nobody waits.

2,048 dishes, a prefill step: the kitchen hand is done at 4.2 ms and waits while the cooks work 29 ms, for about 70,000 tokens a second.

The third case is the moment when, if you put your ear against the server, I am fairly sure you can hear the matrix yelling fuck yeah, pal: for once every box that crosses the bus has a crowd of tokens waiting for it and every tensor core has work. Per token, the decode step costs about 300 times more.

The crossing is the balance point. Memory is a flat line at 4.2 ms, compute a slope of 0.014 ms per token, and they meet at 4.2/0.014≈2954.2 / 0.014 \approx 2954.2/0.014≈295 tokens: below it the step is memory-bound and its length does not depend on the tokens, above it the step is compute-bound and grows 0.014 ms per token.

0100200300400500600tokens in the step02468msmemory rulescompute rules≈ 295 tokensmemory: 4.2 ms, fixed(14 GB ÷ 3.35 TB/s)compute: 0.014 msper tokenthick line: how long the step lasts (the larger of the two)

One step of a 7B in fp16 through its 32 layers on an H100. The crossing is 990 ÷ 3.35, so the size of the model cancels out.

Write the crossing out and the 14 cancels: 14 GB / 3.35 TB/s14×109 ops / 990 TFLOP/s=9903.35≈295\frac{14\ \text{GB} ,/, 3.35\ \text{TB/s}}{14 \times 10^{9}\ \text{ops} ,/, 990\ \text{TFLOP/s}} = \frac{990}{3.35} \approx 29514×109 ops/990 TFLOP/s14 GB/3.35 TB/s=3.35990≈295. The model's size sits above and below the line, so the balance point belongs to the GPU, not the model: a 70B in fp16 on the same card (if it fit) balances at the same 295. In fp16 a weight is 2 bytes and does 2 operations per token, so the operations per byte read, the arithmetic intensity, are exactly the tokens in the step. What moves the point is another GPU or another format: in 4 bits, half a byte per weight, it drops to about 74 tokens.

None of that arithmetic mentioned layers or order, only the step's totals, and it still gave the right time. That is the accounting level, a roofline analysis, which gives a bound; the execution level is what the chip runs, 32 layers in a row of kernels and tiles. They agree here because every layer of the 7B has the same ratio of bytes to operations; in general the real time is the sum of each layer's slower side, never below the bound, plus the cost of launching kernels.

accounting level: totals of the step, no orderbytes14 GB ÷ 3.35 TB/s = 4.2 msops2,048 × 14B ops ÷ 990T/s = 29 msstep = max(4.2, 29) = 29 msexecution level: 32 layers in a rowbytesopslayer 1layer 32each layer: max(0.13, 0.9) ms · 32 × 0.9 = 29 ms One prefill step of 2,048 tokens through a 7B in fp16 on an H100. Both levels give the same answer because every layer has the same ratio; if they did not, the accounting would fall short.

The two phases do the same arithmetic per token with opposite bottlenecks, the most useful diagnosis I know on any machine:

Prefill

Decode

the whole prompt, or a chunk of it Tokens per step one per conversation

every token in the step Each weight tile serves one token

arithmetic (tensor cores) Limited by memory bandwidth

2,048 tokens in 29 ms 7B fp16 on one H100 4.2 ms per token

time to first token (TTFT) Felt as tokens per second after it

prompt eval rate Ollama reports it as eval rate

93 tok/s On the laptop, Qwen3.6 11.7 tok/s

Decode in practice

The first token of the answer exists and the second depends on it, so decode cannot escape one round of the pantry per token, a decoding step with a single column. Every active weight crosses the bus once per token, and that sets a ceiling:

tok/s≤bandwidthbytes read per token\text{tok/s} \le \frac{\text{bandwidth}}{\text{bytes read per token}}tok/s≤bytes read per tokenbandwidth

Double the model, half the speed. On the H100 that is 3,350 / 14, about 240 tokens a second for the 7B; a 13B gives about 129, a 70B about 24 if it fit. On the laptop, 1.84 GB per token from 128 GB/s gives about 70 tokens a second, and the engine reaches 11.7, measured. Real runs never touch the ceiling (this laptop's memory delivers 30 to 40% of its theoretical bandwidth to this access pattern, and each of the 40 layers adds a fixed cost for launching GPU work), but the ceiling explains the ratio between models. The dense 27B reads about 16 GB per token, a ceiling of 8 tokens a second, and measured 2.6: eight times the bytes, about four and a half times slower, which is the whole case for mixtures of experts on a machine like this. A reasoning model's thinking is decode too, one column per thinking token, so the 20,000 thinking tokens of a hard problem cost the 7B on an H100 some 83 seconds before the answer starts.

Other machines get closer. In the 67 published llama-bench runs of dense models that TokenWalk's timing model is fitted on, from an M1 to an MI300X, one stream uses a median of 74% of the spec-sheet bandwidth; the laptop's dense run uses 34%, and the MoE 17%, because its layers read so little that the fixed cost weighs more. A Strix Halo, also an AMD iGPU, reaches 77 to 81% under llama.cpp, so how much of the gap is Ollama on Windows I have not measured.

Share of the specified memory bandwidth one decoding stream uses, computed as tokens per second times bytes read per token over the spec sheet's GB/s. Grey: 67 published llama-bench generation runs of dense models (tg from an empty context) from TokenWalk's data/benchmarks.json, with bandwidths from data/machines.json and data/bench-hardware.json; bytes per token are the file size, and MoE rows are left out because theirs is not. Blue: this laptop's Ollama medians from the apu-llm benchmark, 11.7 and 2.6 tok/s, with bytes per token from the two GGUF files.

This is why quantization matters even when a model fits: halving the bytes roughly doubles the writing speed, while spare memory is worth nothing. The laptop's kitchen has a lower balance point than the H100's, about 18 operations per byte, and a decode step there does about 3.3 (two per weight, at 0.61 bytes per weight), so its cooks also spend most of the step waiting.

One conversation can only put one dish in each box. A server fills the box with other people's dishes instead (batching): 64 users decoded together share one trip of the weights, the step still takes 4.2 ms with their arithmetic only 0.9 ms of it, and the card writes about 15,000 tokens a second with its tensor cores busy 22% of the time. Each of those users still waits 4.2 ms per token, the same 240 tokens a second as alone, so batching raises what the server delivers and leaves each user's speed where it was. Up to about 295 users the step does not get longer, apart from each user's own KV cache.

Prefill in practice

Now the minute of silence. Prefill exists because at the start of a request the whole prompt is known and the cache is empty, so the whole-matrix picture comes back once: all the prompt's tokens go through the layers together, as in training but without the backward pass, with the mask keeping each blind to the ones after it, and every layer is a matrix times a matrix. It leaves every token's keys and values in the cache and the last token's final residual, from which the first word comes. On the laptop Ollama feeds it in chunks of 512 tokens, far past the iGPU's balance point (about 18 operations per byte, which fewer than ten tokens per read already reach). Qwen3.6 needs about 6 billion operations per token, 6,500 tokens are about 39 trillion, and the laptop's GPU manages 0.6 to 0.75 trillion useful operations a second on this work: a little over a minute. A desktop card helps far more here than in decode, because the gap in compute units (8 against 170) is much larger than the gap in bandwidth.

Prefill is one step, and still it grows with the prompt, which the step-time picture can hide. With the whole prompt in one step, the weights part is linear (29 ms for 2,048 tokens on the H100, 450 ms for 32,000), and the attention scores grow with the square of the prompt: about 4% on top at 2,048 tokens and 60% at 32,000, some 0.7 s in all1. In the 7B example with a 2,048-token prompt and a 500-token answer, prefill takes 29 ms and the answer 2.1 s, 98% of the time: prefill brings the boxes once for all its tokens, decode once per token.

1Kaplan et al., Scaling Laws for Neural Language Models, 2020, section 2.1: the attention term stays below the rest of the pass while the context is shorter than about twelve times the model width.

A slow start is prefill, and the levers are on the prompt: fewer and better passages, a summary instead of the whole history, a fixed beginning the engine can reuse. Slow writing is decode, and the lever is bytes per token: a smaller format, fewer active parameters or a wider bus.

Chunked prefill

A server (vLLM, SGLang, TensorRT-LLM) does not push a 32,000-token prompt through as one step. It cuts it into chunks of, say, 512 to 2,048 tokens and runs one chunk per step; each chunk writes its keys and values into the cache, and the last one produces the first token of the answer. The result is identical to doing it all at once, thanks to the causal mask: chunk 2's tokens attend to themselves and to chunk 1's keys and values read from the cache, and a token looking backwards cannot tell whether what is behind it was computed a moment ago or cached (the only extra cost is re-reading earlier chunks' keys and values). It is a decision of the server (tiling, by contrast, happens inside every step), and there are three reasons for it:

A 32,000-token prompt in one step holds the GPU for almost half a second, and every user who was in the middle of an answer stalls.

A decode step leaves the cooks idle while the boxes come anyway, and a chunk of prefill can ride on that same round (piggybacking).

Intermediate results grow with the tokens in the step, and attention's with their square, so chunks cap the memory peak.

The ride is free only up to the gap. Fifty users decoding take 4.2 ms with the SMs busy 0.7 ms of it, so about 245 more tokens fit before the step reaches the 295 of the balance point and compute takes over.

50 users decoding4.2 ms · SMs busy 17%busSM50 + 245 prompt tokens4.2 ms · the prefill rides freebusSM50 + 1,000 prompt tokens≈ 15 ms · the 50 users wait toobusSMthe same in two separate steps4.2 + 14 = 18.2 ms · two tripsbusSMbus: weightsSMs: 50 decodeSMs: prefillwait

One step of a 7B in fp16 through its 32 layers on an H100, all four on one time axis. The prompt rides free up to the balance point, 295 tokens in the step; past it, the step grows 0.014 ms per token for everyone.

Put 1,000 prompt tokens into that step and it lasts 1,050×0.014≈151{,}050 \times 0.014 \approx 151,050×0.014≈15 ms: still one round of the pantry saved against a separate step (4.2 + 14 = 18.2 ms), but the fifty decoding users pay with a 15 ms step instead of 4.2, a visible stutter in their text, so the chunks mixed into decode are moderate. Chunks above about 300 tokens add no speed either: 32,000 tokens in 16 chunks of 2,000 make 16 rounds (67 ms) hidden under 450 ms of arithmetic, the same as all at once.

The KV cache in bytes

The weights are not all the kitchen hand carries. In decode, each conversation's KV cache is a second sack, read whole at every step over the same bus, so its size is a speed question too:

MKV=2×nlayers with KV×nkv heads×dhead×b×ntokensM_{KV} = 2 \times n_{\text{layers with KV}} \times n_{\text{kv heads}} \times d_{\text{head}} \times b \times n_{\text{tokens}}MKV=2×nlayers with KV×nkv heads×dhead×b×ntokens

For Qwen3.6 in 16-bit numbers that is 2×10×2×256×2=20,4802 \times 10 \times 2 \times 256 \times 2 = 20{,}4802×10×2×256×2=20,480 bytes, 20 KB per token: grouped-query attention divided it by eight, and keeping it in ten layers of forty by four. The plain 7B of the H100 example, 32 layers with keys and values for every head across a width of 4,096, keeps 2×32×4,096×22 \times 32 \times 4{,}096 \times 22×32×4,096×2 bytes, 0.5 MB per token, twenty-five times more, and every decode step moves the weights plus all of it.

tok/s0 tok14 GB per step2392,500 tok14 + 1.3 GB per step2198,000 tok14 + 4.2 GB per step18416,000 tok14 + 8.4 GB per step15032,000 tok14 + 16.8 GB per step109weights, 14 GBKV cache, 0.5 MB/token

One decode step for one user of a 7B in fp16 without GQA, on an H100. Every decode step reads the weights and the whole cache over the same bus, so the speed is 3,350 GB/s divided by the bytes per step.

At the end of a 2,000-token prompt with a 500-token answer the step moves 15.3 GB and writes about 219 tokens a second instead of 240. At 32,000 tokens the second sack weighs about 17 GB, more than the model, the step moves twice the bytes and the speed halves to about 110 tokens a second. Grouped-query attention with 8 KV heads, as in Mistral 7B, would divide the cache by four, and batching does nothing for it, since each conversation brings its own sack.

Running it on the laptop

Back to the desk, where the memory is shared with Chrome and every setting is mine to get wrong. The physics is the H100's with smaller numbers, and the engine makes choices for you.

Context reservation and the memory budget

The cache is reserved at load time for the whole context the request asks for (num_ctx), used or not, and Ollama sizes that default from the memory it detects: here it chose 262,144 tokens, 5.4 GB reserved for a 7,000-token conversation, four times that for a model without the hybrid trick. Setting OLLAMA_CONTEXT_LENGTH to what the application needs is the first change I would make (the console with the full reservation shows what it holds back). Before any model loaded, Windows with Chrome, Cursor, Claude Code and Anki was already using 32 GB, and Chrome, as usual, was having the time of its life.

num_ctx = 16,384 (set by hand)32 GB21.7 GBKV 336 MBfree ≈ 9.8 GBnum_ctx = 262,144 (Ollama chose it)32 GB21.7 GBKV 5.4 GBfree ≈ 4.8 GBsystem and appsweightsKV cachefree

Each bar is the laptop’s 64 GB, to scale, with Qwen3.6 in Q4_K_M and its cache in f16. The cache is reserved for the whole num_ctx, used or not; without the hybrid layers it would take 4 × 5.4 = 21.5 GB.

With Qwen3.6 and 16,384 tokens of context about 10 GB stay free, about what the system needs to breathe; on 64 GB the problem is speed, never fitting. The cache can be stored in fewer bits, separately from the weights (KV cache quantization, OLLAMA_KV_CACHE_TYPE=q8_0, about half the memory): quantizing the weights rounds the model, quantizing the cache rounds the patient's history as the model has read it. Ollama requires the fused FlashAttention kernel (OLLAMA_FLASH_ATTENTION=1) for a quantized cache.

Tricks for faster decoding

Every trick for faster writing trades cheap work or cheap memory for expensive reads, and each has a condition under which it stops paying. Three of them decide what this laptop feels like: checking several guessed tokens per read of the weights, not reading a prompt twice, and putting less text in the prompt to begin with.

Speculative decodingPrefix cachingRetrieval

Speculative decoding gives decode some of prefill's reuse: a small draft model with the same tokenizer (Qwen3 0.6B for a Qwen) writes a few tokens ahead, and the big model checks them all in one pass, a tiny prefill that costs about one ordinary step. It is an intern drafting and a boss signing, and the boss signs from the left until the first token it disagrees with, where it puts in its own. A rejection rule keeps the output distribution exactly the big model's1, so only the speed changes, and it depends on the acceptance rate α\alphaα. With γ=4\gamma = 4γ=4 draft tokens the first of a round is always produced, the second needs one acceptance, the third two in a row, and if all four pass the same pass yields a fifth:

1Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023, which proves the output distribution is preserved and derives the expected speedup.

E[tokens per round]=1−αγ+11−αE[\text{tokens per round}] = \frac{1 - \alpha^{\gamma + 1}}{1 - \alpha}E[tokens per round]=1−α1−αγ+1

At α=0.7\alpha = 0.7α=0.7 a round gives 2.8 tokens for one pass of the big model. For our two models the verdict comes out opposite, which is the part I found most interesting. The dense 27B reads 16 GB once whether it checks one token or five, and four steps of a 0.6B draft are cheap next to that, so a round is roughly twice as fast. The MoE's ordinary step reads only 1.84 GB, while a check of five tokens wakes up to forty experts per layer instead of eight, so the check reads several times more and the draft costs more than it saves. That comes from first-order costs; measuring it is the experiment to run (dense with a draft, MoE with the same draft).

Qwen3.6's file carries a built-in draft, a 41st block for multi-token prediction that Ollama runs with two drafts per round, and on this laptop it only pays near greedy sampling: at temperature 0 it lifted writing from 13.2 to 15.6 tokens a second, and at 0.7 it cost time (11.5 and 10.6 against 13.2), because a cheaper guess gets rejected more often.

A model remembers nothing between requests. A chat feels continuous because the application resends the whole conversation every time, the context window of the call, and every call pays prefill on all of it unless the engine has seen the beginning before. That is prefix caching: call 4 begins with the same tokens as call 3, so by the causal mask their keys and values are identical, and an engine that kept them computes only the new tail. The match must be exact from the first token. Change one word of the system prompt and every later token attended to it, so call 4 recomputes all 7,600 tokens and pays the full minute again; stable material goes first and the variable question last.

Anthropic charges 1.25 times the input price to write a prefix into its five-minute cache and a tenth or less to read it1. Ollama keeps the last request's cache in the loaded model, which I found by accident when a benchmark reused a prompt opening and read 2,900 tokens at 7,000 to 9,000 tokens a second. llama.cpp's server can also save a slot's cache to disk (--slot-save-path): the 6,500-token history costs about 70 seconds of prefill and 133 MB of cache, which an SSD reads back in a tenth of a second, so a copilot that restores each patient's cache answers in about a second (for a hybrid model the DeltaNet state must be saved too, which I have not tested).

1Anthropic, Prompt caching, pricing and the invalidation order tools, system, messages.

Most long prompts come from retrieval-augmented generation: passages relevant to the question are found and pasted into the prompt, usually by hybrid search, which merges BM25 (exact words, good with drug names, blind to synonyms) and an embedding model (finds oral anticoagulant for warfarin, can miss an exact code). Retrieval decides how much text enters prefill: at about 90 tokens a second, ten passages of 500 tokens add almost a minute before the first word, so fewer, better passages are the cheapest speed-up a RAG system has.

The laptop runs a 35B model privately and for free at about reading speed, and is the wrong machine for long runs of arithmetic (an evaluation over thousands of cases, a fine-tune). For those I rent an RTX 5090, $0.69 an hour on RunPod's community cloud in October 20261, as I did to fine-tune Stable Diffusion.

1RunPod, Pricing, page updated 8 October 2026.

Ollama and the model lifecycle

The model itself can leave memory. Ollama is a background server on localhost:11434, the ollama command one of its clients, and a model goes through five states with it.

1Pullonce, network time2Load55 s cold, disk to RAM3Serveprefill and decode4Idleholds memory5Unloadafter keep_alive

Keep-alive decides the last step, five minutes by default, and the first request after an unload pays the whole load again: ask every three minutes and the model is always there, ask every ten and each question waits 55 seconds before prefill even starts (a cold start in the console). These are the settings the laptop needed, plus one that made the GPU usable at all:

OLLAMA_IGPU_ENABLE1without it Ollama ignores the 860M and runs on the CPU OLLAMA_CONTEXT_LENGTH16384default num_ctx; stops the 262,144-token reservation OLLAMA_FLASH_ATTENTION1blockwise attention, required for a quantized cache OLLAMA_KV_CACHE_TYPEq8_0cache at about half the memory OLLAMA_KEEP_ALIVE30mavoids reloading between bursts of use

The server reads them at start, so they need a restart (setx and a restart from the tray on Windows, systemctl edit ollama on Linux; export in a terminal does nothing to a running server). Options sent in a request override them for that call, and every response reports the three numbers this article is about:

PYTHON Copy

import requests r = requests.post("http://localhost:11434/api/generate", json={ "model": "qwen3.6:35b-a3b", "prompt": open("history.txt").read(), "stream": False, "options": {"num_ctx": 16384}, }).json() s = lambda k: r[k] / 1e9 # durations come in nanoseconds print("load", s("load_duration")) print("prefill tok/s", r["prompt_eval_count"] / s("prompt_eval_duration")) print("decode tok/s", r["eval_count"] / s("eval_duration"))

Ollama hides the engine's options and manages files and memory; llama.cpp's server exposes everything (a draft model with --model-draft, caches on disk) and manages nothing; LM Studio sits between them.

Numbers and answers

Most questions about whether a model will run, and how fast, can be answered before downloading anything, and the rest of the questions I had after abandoning the class fit in a table.

Back-of-the-envelope arithmetic

Size GB of a modelparameters in billions × bits per weight ÷ 835.5 × 4.9 ÷ 8 = 21.7 Bits per weight16 FP16, 8.5 Q8_0, ~4.9 Q4_K_MQ4_K_M is about 0.6 bytes Memory neededweights + KV cache + ~0.5 GB + 8 to 12 GB for the system Speed Decode ceilingbandwidth ÷ GB read per token128 ÷ 1.8 = 70 tok/s Decode, real30 to 50% of the ceiling on this laptop11.7 measured; ~75% in published dense runs Prefilluseful TFLOPS × 1e12 ÷ (2 × active parameters)0.6e12 ÷ 6e9 = 100 tok/s TTFTprompt tokens ÷ prefill tok/s6,500 ÷ 93 = 70 s Answer timeoutput tokens ÷ decode tok/s300 ÷ 11.7 = 26 s Step timethe larger of bytes ÷ bandwidth and operations ÷ FLOP/snever the sum; 7B decode on an H100: 4.2 ms Balance pointFLOP/s ÷ bandwidth, in tokens for fp16~295 on an H100, whatever the model size Cache KV per token2 × KV layers × KV heads × head size × bytes20 KB for Qwen3.6, 0.5 MB for the 7B Text 1,000 English words~1,300 tokensSpanish 1,300 to 1,600 Reading speed5 to 8 tok/sanything above it feels instant

The one I use most is the decode ceiling. A model that fits but writes below reading speed is a model you will stop using: on this laptop, 60 GB of dense weights fit and would write about two tokens a second.

Short answers

Question Short answer

Do I need attention and Q, K, V to be an applied AI engineer? As mechanism, yes: the KV cache and everything built on it follow from them. The derivations, no.

Are intermediate results kept? Activations are overwritten layer by layer; only keys and values at attention layers are kept. Queries never are.

Is the KV cache only for the first layer? It is per attention layer, each from that layer's residual; in Qwen3.6 the 10 full-attention layers of 40.

Is the whole model ever on the GPU's cores? Never: it sits in HBM (or the laptop's RAM) and crosses to the cores in tiles of about 32 KB, every step.

Did I buy a bad computer? You bought capacity over speed: it holds models a 16 GB card cannot, at reading speed.

Does memory swap experts per token? No, all 256 stay in memory; 8 run per token here, 2 of 8 in Mixtral.

Trace of the request

Back to the question at the top, with everything in place.

1Ollama receives the messages over HTTPlocalhost:114342The chat template turns them into one ChatML string3The tokenizer cuts it into about 6,550 token IDs4Each ID becomes its row of the embedding table2,048 numbers5Prefill runs 13 chunks of 512 through all 40 layersabout 70 s, the GPU's 8 CUs busy6The ten attention layers store keys and values134 MB of KV cache7The last column is scored against 248,320 tokensthe LM head8The sampler draws the first token9Each next token is one decoding stepreads ~1.9 GB, ~85 ms10After about 300 steps the end-of-turn token stops the loop~26 s of writing

The minute of silence was the whole-matrix picture computed once, on 8 compute units with plenty of tokens per weight. The twelve tokens a second were one scrawny column reading 1.84 GB per token from a 128 GB/s bus. Almost every part of this pipeline existed before 2017, attention included; what the transformer changed was the order in which work can happen, every position at once in training, which is why it scaled. Generation is the one place where that parallelism ends, and every trick in the second half of this article makes that one sequential column read fewer bytes per token or share them with someone else.

11.7 tok/sQwen3.6 35B-A3B writing on the Radeon 860M, against a bandwidth ceiling near 70the measurements of the previous article 93 tok/sthe same model reading a 2,900-token promptsame run 2.6 tok/sthe dense Qwen3.8 27B writing, which reads about 16 GB per tokensame run 4.2 msone step of a dense 7B in fp16 on an H100, the time to stream 14 GB at 3.35 TB/s, for 1 token or 295NVIDIA's spec sheet, cited above 20 KBKV cache per token of Qwen3.6 in f16, from 10 attention layers of 40computed from the GGUF header 4.89 bitsaverage bits per weight of the 21.7 GB filesummed from its 753 tensors