Transformers and local inference

A course from zero on the transformer and on running one locally, built around a single request: a clinical question over a 6,000-token synthetic patient history, sent to Qwen3.6 35B-A3B on a laptop with a Radeon 860M and 64 GB of shared memory. It starts from one dot product and adds one idea at a time, until the two phases of inference and their bottlenecks follow from what came before. The second half sizes the KV cache with the real numbers of a hybrid model and works out when each trick for faster decoding stops paying. Eighteen small simulations, each with one control, sit between the steps, and a standalone simulator follows the whole request.

Credit26 · oct 23Read notethe banner is a frame from 3Blue1Brown's visual walkthrough of transformers, animated with Manim, Grant Sanderson's open-source math animation engine, and still the best place to build intuition for attention. The transformer half started as my notes on it and grew into an earlier transformers article, which this one absorbs and replaces. The running half comes from a class I prepared for myself on local models and abandoned halfway through, and from the questions I could not answer when I tried to finish it.

I have a laptop on my desk and a question for it. The question is the kind a clinical copilot would get, any interaction with the new anticoagulant?, asked after a short system prompt and the history of a patient (synthetic, about 6,000 tokens of it, which is roughly a 4,000-word document). The laptop runs a language model locally, so the patient's data never leaves the machine. I press enter, and for a little over a minute nothing happens. Then the answer starts, and it comes out at about twelve words a second until, some 300 tokens later, it stops.

Those two numbers, the minute of silence and the twelve tokens a second, are what this article is about. I want to be able to say where each one comes from and what it would cost to change it. To get there I am going to start from the smallest possible piece (one multiplication of two lists of numbers) and add one idea at a time, each one resting on the ones before. If you know what a list of numbers is, you can follow it. There is a small interactive figure after most steps, each with a single thing to touch, and they all use the same laptop and the same model, so the numbers agree from one figure to the next.

The laptop is a Lenovo ThinkPad with an AMD Ryzen AI 7 PRO 450 and 64 GB of memory; its only graphics processor is the Radeon 860M built into the same chip1. The model is Qwen3.6 35B-A3B, served by Ollama. The speeds come from a day I spent measuring this laptop, and the model's details from its own file, which I opened byte by byte. For the impatient, the standalone simulator follows the whole request from the disk to the last token; everything in it is explained below, one piece at a time.

1AMD, Ryzen AI 7 PRO 450 product page: 8 cores (4 Zen 5, 4 Zen 5c), 16 threads, boost up to 5.1 GHz, NPU up to 50 TOPS.

The dot product

Everything a language model does in bulk is one operation, repeated billions of times. Take two lists of numbers of the same length, multiply them position by position, and add up the results. That is a dot product. With x=(0.8,0.1,0.5)x = (0.8, 0.1, 0.5)x=(0.8,0.1,0.5) and w=(0.31,−1.2,0.07)w = (0.31, -1.2, 0.07)w=(0.31,−1.2,0.07) it is 0.8⋅0.31+0.1⋅(−1.2)+0.5⋅0.07=0.1630.8 \cdot 0.31 + 0.1 \cdot (-1.2) + 0.5 \cdot 0.07 = 0.1630.8⋅0.31+0.1⋅(−1.2)+0.5⋅0.07=0.163.

The way I like to read it is as a question: how much does xxx look like www? If you draw the two lists as arrows, the dot product is large and positive when they point the same way, about zero when they are perpendicular, and negative when they point in opposite directions. Two numbers per arrow are enough to see it, and the same thing holds with three numbers, or with 2,048.

x · w at 35°0.82 A dot product as two arrows: the closer their directions, the larger the number; perpendicular arrows give zero, opposite ones a negative number.

Both arrows here have length one, so the dot product is just the cosine of the angle between them; the cyan bar is how much of www lies along xxx. A neuron, in the sense people use for neural networks, is exactly this: a dot product of its inputs with its weights, usually followed by a function that cuts off the negative part. The weights are the arrow www, and they are what training chose. When people say a model has 35 billion parameters, they mean 35 billion of these numbers (the B is the English billion, a thousand million).

Layers as matrices

One neuron asks one question. Put many neurons side by side, all reading the same input, and you get a matrix: each row is one neuron's weights, and multiplying the matrix by an input vector computes every row's dot product at once. A matrix of 4,096×4,0964{,}096 \times 4{,}0964,096×4,096 holds 16.7 million weights and turns a vector of 4,096 numbers into another of 4,096. A language model is a few hundred such matrices applied one after another, and running the model means multiplying vectors by them, over and over.

Here is the fact the whole second half of this article leans on, so I want to see it on a toy before it matters. To multiply one vector by a matrix, every weight has to be fetched from memory once, and then it is used for exactly one multiplication and one addition. Now multiply sixteen vectors by the same matrix. Every weight is still fetched once, but each one is used sixteen times.

weights read48 multiply-adds for one token48 A 6 by 8 weight matrix read once and multiplied by one or more token columns: the reading stays at 48 weights, the multiply-adds grow with the tokens.

The reading stays at 48 weights no matter how many columns you add; the arithmetic grows with them. Keep this picture in mind, because a model that writes text one token at a time is the case on the left of that slider, with a single column. Whether a computer is busy multiplying or busy waiting for numbers to arrive depends on how many columns share each read, and we will come back to it with real numbers once we have a real machine.

For counting, one weight costs one multiplication and one addition, so a token passing through a model costs about two operations per weight it touches.

FLOPs per token≈2×Pactive\text{FLOPs per token} \approx 2 \times P_{\text{active}}FLOPs per token≈2×Pactive​

For our model that is about 6 billion operations per token (it touches about 3 billion weights per token, for reasons we will get to with the experts).

Tokens and embeddings

A matrix wants a vector, and we have text. The model never sees letters: a tokenizer cuts the text into pieces from a fixed vocabulary and hands the network one integer per piece. Common words are one piece, rare ones split into several, and a leading space usually belongs to the word after it. Qwen3.6's vocabulary has 248,320 entries (its file lists them all; the first six are !, ", #, $, % and &).

The pieces are learned once, before training, by BPE, and the algorithm is simple enough to run in a browser. Start from single characters (or bytes), find the pair of neighbours that appears most often in a corpus, glue them into a new entry, and repeat until the vocabulary is as large as you want. Qwen's file stores its 247,587 merges in order, and the first ones are two spaces, four spaces, i n and space t. The figure below trains one on a single clinical paragraph while you watch; type anything and see how it gets cut.

characters63 UTF-8 bytes63 words12 tokens25 vocabulary161 A byte-pair tokenizer trained on one paragraph, keeping 120 merges on top of 41 characters, cuts the sentence into 25 pieces: T | he | ·patient | ·present | s | ·with | ·f | ev | er | ·and | ·a | ·dr | y | ·co | ug | h | ·f | or | ·th | re | e | ·d | ay | s | .. As characters it would be 63 pieces, and a model writes one piece per pass through all its layers.

It learns that paragraph's words quickly and splits everything else into small pieces, because one paragraph is all it has seen; Qwen's was trained on terabytes. It might seem simpler to use letters, and it would be, but it costs more. English and Spanish average about four characters per token, so a character model would handle four times as many positions, and (more painfully on a laptop) it would write four times as many of them. Each written position is a full pass through the model, so a 300-token answer would become about 1,200 passes for the same words. For budgeting, 1,000 English words are about 1,300 tokens and 1,000 Spanish words 1,300 to 1,600.

A chat is a list of messages and the model only continues text, so a chat template flattens the list into one string, with special tokens marking who speaks. The template ships inside the model file; Qwen uses the ChatML layout, and the model writes from the last line on:

TEXT Copy

<|im_start|>system You are a clinical assistant. Cite the record.<|im_end|> <|im_start|>user Any interaction with the new anticoagulant?<|im_end|> <|im_start|>assistant

So the system prompt is nothing more mysterious than the first lines of a long text. Everything else a request carries, images included, ends up as tokens in this same string.

Each token ID then becomes a vector by a plain lookup. The embedding matrix has one row per vocabulary entry, 2,048 numbers wide in Qwen3.6, and token 4,821 simply takes row 4,821. At the start of training the rows are noise; training nudges them until related tokens point in related directions, which is all an embedding is. The table is large but nearly free to use, since a token only reads its own row.

One problem remains. Looked up one by one, dog bites man and man bites dog give the same three vectors, and nothing so far knows about order. Positional encoding fixes that. Current models, Qwen among them, use RoPE, which rotates parts of each vector by an angle proportional to its position, so that when two of them are compared later, the result depends on how far apart they are. Qwen3.6's file records a rotation base of ten million, part of how it reaches a context of 262,144 tokens.

Attention

After the lookup we have one column of 2,048 numbers per token, and these columns are never thrown away. Every block of the network reads them and adds a correction back on top (a residual connection), so each token's column is a running sum that every layer writes into, the residual stream. The first kind of block lets the columns exchange information.

Take the word model. It has one embedding, and it has to mean one thing in a machine learning model and another in a fashion model. Attention is the step that lets machine learning write into the column of model, so that a few layers later the column knows which one it is.

The mechanism is three more matrices and the dot product we already know. Each attention head multiplies every token's column by three learned matrices, WQW_QWQ​, WKW_KWK​ and WVW_VWV​, giving each token three short vectors (query, key and value, 256 numbers each in Qwen3.6). I think of them as a little marketplace, where the query is what a token is looking for. The key is what it advertises, and the value is what it hands over if chosen. Every token's query is dot-producted with every key (the question again: how alike are these?), the scores go through a softmax that turns them into weights between 0 and 1 that sum to 1, and the token's update is the weighted sum of the values.

Here is that on a short sentence, with queries and keys I wrote by hand so that the pronoun has someone to find. Pick a word and look at where its weight goes.

strongest pullwarfarin The attention weights of the pronoun its over the words before it: most of its weight goes to warfarin.

From its, most of the weight lands on warfarin, which is exactly what you would want the word to absorb. A real head learns its vectors instead of having them written by hand, and runs with 256 numbers per vector instead of a handful, but the arithmetic is the same. In one formula, with dkd_kdk​ the length of a key (dividing by its square root keeps long vectors from producing huge scores that would make the softmax all-or-nothing):

Attention(Q,K,V)=softmax⁡ ⁣(QKTdk)V\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right) VAttention(Q,K,V)=softmax(dk​​QKT​)V

You may have noticed that in the figure no word ever looks to its right. That is on purpose, and it turns out to be the most consequential rule in this whole article. In a model that writes text, a token may attend to itself and to earlier tokens, never to later ones: every score where a query meets a later key is set to minus infinity before the softmax, and e−∞=0e^{-\infty} = 0e−∞=0. This is the causal mask. It exists for training (one sentence of eight words gives eight prediction tasks in one pass, and none of them may peek at its own answer), and it has a consequence you can see by writing the sentence one word at a time.

rows computed per new word1 Attention weights of a sentence written word by word: future words are masked, and adding a word adds one row while the rows above stay exactly as they were.

Add a word and exactly one new row appears. The rows above it do not change by a single digit, because none of those words was allowed to look at the new one. A token's column at every layer depends only on itself and the tokens before it, so appending a token never changes anything that came earlier. Everything we work out about earlier tokens stays correct forever and can be kept. Hold on to that; it is the reason the KV cache exists.

One head captures one kind of relation, so each layer runs several side by side (multi-head attention) and adds their updates. Qwen3.6 has 16 query heads per attention layer but only 2 sets of keys and values, each shared by 8 heads. That sharing is grouped-query attention, and it will matter for memory: it stores eight times fewer keys and values than giving every head its own, at a small cost in quality1.

1Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023, which found the grouped form close to full multi-head attention in quality at nearly the speed of a single shared set.

The MLP and its experts

After attention, every column goes on alone through an MLP: two (in current models three) matrix products with a nonlinearity between them. You can read the first matrix row by row as a list of patterns, each one asking how much the column matches it; the nonlinearity zeroes the ones that scored low; the second matrix adds a direction to the column for each pattern that fired. A pattern and the direction it adds make a small stored fact, and an MLP holds tens of thousands of them per layer, which is where most of what a model knows seems to live1. In an ordinary (dense) model the MLPs hold about two thirds of the parameters, and every token goes through all of them.

1Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories, 2021; Meng et al., Locating and Editing Factual Associations in GPT, 2022, edited single facts by changing MLP weights in middle layers.

A mixture of experts splits that big MLP into many small ones and adds a router that sends each token to a few. Qwen3.6 has 256 experts in each of its 40 layers, each one three matrices of 2,048×5122{,}048 \times 5122,048×512. A router (one more small matrix) scores all 256 for the current token, the top 8 run, and one shared expert runs for every token. That is where the name comes from: 35 billion parameters in total, about 3 billion active per token, 35B-A3B.

experts per written token8 + 1 shared The 256 experts of one Qwen3.6 layer, lit by the tokens of one pass: 8 for a written token, many more for a 512-token prompt chunk.

With one token in the pass, 8 experts per layer wake up and the rest sleep. Slide to a prompt chunk of 512 tokens and almost all of them wake, because 512 tokens pick different experts. Every expert stays in memory all the time, since the next token may pick any of them, so a mixture of experts saves nothing in capacity; it saves on what is read and computed per token. And that saving is large exactly when one token is in flight, which is when the model is writing. I picked the routing with a seeded random generator that favours some experts over others, as real routers do, which is why it touches a few fewer than the independent-pick formula.

Two smaller pieces finish the layer. A normalisation rescales each column before a block reads it, so the numbers neither explode nor fade over 40 layers; it weighs almost nothing. And the surprise in Qwen3.6's file is the key full_attention_interval = 4: only one layer in four is the attention we just saw (ten layers out of forty). The other thirty are Gated DeltaNet layers, a form of linear attention that keeps a fixed-size state per layer, updated by each token the way a recurrent network does1. A fixed state cannot reach back to one specific earlier token the way attention can, which is why some full-attention layers stay. Remember the number ten; it comes back when we count memory.

1Yang, Kautz and Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025.

Next-token sampling

After the last layer, the final token's column is multiplied by one more matrix, the LM head, which has one row per vocabulary entry. That is 248,320 dot products, one score per possible next token, called logits. Scores can be any real number, so a softmax turns them into probabilities: exponentiate each one (everything becomes positive, and the gaps stretch) and divide by the total. Before that, every logit is divided by a temperature TTT.

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}pi​=∑j​ezj​/Tezi​/T​

That division is the whole of temperature, and it is easier to feel than to describe. Here are seven candidates for the word after lower the warfarin, with logits I made up.

top token at T = 1about 70% Seven candidate next tokens and their probabilities under a temperature slider: lower sharpens toward the favourite, higher flattens, the order never changes.

The dashed outlines stay at temperature 1 for comparison. Below 1 the favourite takes over; above 1 the distribution flattens and even banana gets a chance. The order of the candidates never changes, and the probabilities always add up to one. At the limit of zero the top token takes everything, which is greedy decoding.

Top-p sampling does something different. It sorts the candidates, keeps the smallest set from the top whose probabilities add up to at least ppp, throws the rest away and renormalises the survivors. Holtzman and colleagues proposed it after noticing that sampling from the full distribution occasionally picks a token from the long tail of nearly impossible ones, and that one such token can derail the rest of the text1.

1Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020.

kept at p = 0.93 of 7 The same candidates under a top-p cut: the smallest set from the top reaching p is kept and renormalised.

So temperature reshapes the whole distribution and leaves the tail alive, and top-p removes the tail and leaves the ratios among the survivors alone. Both reduce surprise, in different ways, and models ship with both set (Qwen3.8's file recommends temperature 1.0, top-p 0.95 and top-k 20, where top-k keeps a fixed number of candidates). The sampler is a few lines in the engine, outside the network. A natural shortcut would be to compute only the top few logits, and it does not work: nobody knows which ones are the top until all 248,320 dot products are done.

A small aside on Claude: Anthropic's current models no longer accept temperature, top-p or top-k at all. Since Claude Opus 4.7 a request that sets them returns an error, and from the API version of 21 October 2026 that holds for every model1. Their stated reason is that prompting is the way to steer these models. Mine, which is a guess: a model tuned with its sampler fixed behaves worst exactly when a user changes the distribution it was tuned on.

1Anthropic, API version 2026-10-21 migration guide, change 1, and the Opus 4.7 migration notes, which recommend prompting to guide behaviour and note that temperature 0 never guaranteed identical outputs.

Training

Everything so far turns weights into a prediction; training is the loop that chooses the weights. Feed a batch of text and predict the next token at every position. How wrong the predictions were is measured as one number, the loss. Backpropagation then works out for every parameter which way it should move to reduce that number, and each one moves a small step that way. The loss is cross-entropy: for each position, take the probability the model gave to the token that actually came next and add up its negative logarithm, so a confident wrong guess costs the most. The causal mask is what makes this fast, since every position of every sequence is a separate training example inside one parallel pass. GPT-3 was trained on 300 billion tokens in batches of 3.2 million, about 94,000 steps1.

1Brown et al., Language Models are Few-Shot Learners, 2020, table 2.1.

Pre-training gives a model that continues text well and follows instructions badly. Fine-tuning is the family of fixes, from RLHF to LoRA, and the piece on how LLMs learn covers it. For running a model, three facts are enough. Inference never changes the weights, so the model learns nothing from your questions and knows about your patient only what is in the request. Fine-tuning needs several times the memory of inference, which is why it goes to the cloud. And the tokenizer is never changed after training, since every row of the embedding table belongs to one of its pieces.

The GGUF file

So far a model has been matrices on paper. On my disk it is one file. When Ollama downloads qwen3.6:35b-a3b it stores a few blobs under ~/.ollama/models/blobs, and the largest one, 21,718,480,960 bytes, is the model: a GGUF file, the format of llama.cpp. Opening it in a text editor shows noise, but the layout is simple enough to read with forty lines of Python, and doing that once is the fastest way I know to stop thinking of a model as something mysterious.

GGUF file, 21.7 GBone blob in Ollama's store headerthe four bytes GGUF, version 3, 753 tensors, 36 metadata keys

metadataarchitecture, layers, heads, context, tokenizer, chat template

tensor tableeach tensor's name, shape, number format and offset

tensor datathe weights themselves, aligned, almost all of the size

The metadata is a dictionary, and every number we have used so far is in it:

TEXT Copy

general.architecture = qwen35moe qwen35moe.block_count = 41 # 40 layers + 1 for multi-token prediction qwen35moe.embedding_length = 2048 # width of the residual stream qwen35moe.attention.head_count = 16 # query heads qwen35moe.attention.head_count_kv = 2 # key-value sets (grouped-query attention) qwen35moe.attention.key_length = 256 qwen35moe.full_attention_interval = 4 # 1 layer in 4 is full attention qwen35moe.context_length = 262144 qwen35moe.expert_count = 256 qwen35moe.expert_used_count = 8 qwen35moe.expert_feed_forward_length = 512 tokenizer.ggml.tokens = [248320 entries: '!', '"', '#', '$', ...] tokenizer.ggml.merges = [247587 entries: 'Ġ Ġ', 'ĠĠ ĠĠ', 'i n', 'Ġ t', ...] tokenizer.chat_template = {%- set image_count = namespace(value=0) %} ...

The tensor table names every matrix, and reading the names is reading the architecture. Layer 0 is a Gated DeltaNet layer, layer 3 a full-attention one:

TEXT Copy

token_embd.weight [2048, 248320] Q4_K the embedding table blk.0.attn_qkv.weight [2048, 8192] Q6_K DeltaNet projections blk.0.ffn_gate_inp.weight [2048, 256] F32 the router blk.0.ffn_up_exps.weight [2048, 512, 256] Q4_K 256 experts in one tensor blk.0.ffn_up_shexp.weight [2048, 512] Q4_K the shared expert blk.3.attn_q.weight [2048, 8192] Q4_K full attention: Wq blk.3.attn_k.weight [2048, 512] Q4_K Wk, 2 heads x 256 blk.3.attn_v.weight [2048, 512] Q6_K Wv output.weight [2048, 248320] Q6_K the LM head

Adding up the shapes gives 35,505,251,456 parameters. The file is 21.7 GB, though, and if each parameter took the two bytes of a 16-bit float it would be 71 GB. The difference is the column of formats on the right, Q4_K and Q6_K.

A weight leaves training as a 16-bit floating-point number. Quantization stores it with fewer bits, and the first thing I got wrong about it is that it cannot work by cutting decimals off each number: 4 bits are sixteen possible values, full stop. What it does is take a block of weights (32 of them, say), find one scale for the block, and store each weight as the index of the nearest of the evenly spaced levels that scale allows. Reading a weight back is one multiplication, level times scale.

levels at 4 bits16 A block of 32 weights rounded to the levels one shared scale allows, at 8, 4, 3 or 2 bits, with the error of each weight.

At 8 bits the stored bars sit on top of the dots and the error is invisible. At 4 bits there are sixteen levels and the error is a few percent, small enough that most people run models this way. At 2 bits the block has four levels to share, and the error jumps. Notice also the one large weight in the block: it forces a large scale, and the small weights then have only a few levels near zero to choose from. That outlier problem is what the K formats soften, with a second level of smaller scales inside each block.

Each block also stores its scale, which is why the bits per weight are never round numbers: Q8_0 is 8 bits per weight plus one 16-bit scale per 32 weights, 8.5 bits. Qwen3.6's file mixes Q4_K at 4.5 bits and Q6_K at 6.56 (the matrices that hurt most when rounded keep more bits), and averages 4.89 bits per weight. That is what Q4_K_M means in practice, and the arithmetic of a model's size is always the same: parameters times bits per weight divided by eight, 35.5 billion times 4.89 divided by 8, 21.7 GB.

Parameters and gigabytes are different numbers.

A slide I wrote for myself said the dense 27B model reads 17 GB per token, and I read it as an error, since the model has 27 billion parameters. Both are right: 27.3 billion parameters at 4.9 bits are 16.8 GB. My own notes had the converse mistake, a 20B model at 17 GB in Q4; a 20-billion-parameter model in Q4_K_M is about 12 GB.

Nothing in the file is executable code. The engine supplies the arithmetic; the file supplies the numbers and the shapes. That is why the same file can be copied or hard-linked between engines that read GGUF, which is what the previous article did between Ollama and LM Studio.

Memory capacity and bandwidth

To compute with a weight, a processor needs it in a register on the chip. The chip has only a few megabytes of fast memory of its own, so a 21.7 GB model lives in RAM, and every weight a token needs has to cross the bus from RAM to the chip, for every token. Two properties of that memory matter, and they are easy to mix up. Capacity is how much it holds: 64 GB here. Bandwidth is how fast it can deliver what it holds to whoever computes: 128 GB/s at most on this laptop.

Capacity decides whether a model fits. Bandwidth, it turns out, decides how fast it writes. Here are the two models I have on this laptop, the mixture of experts and a dense one of similar size, measured both ways on the same axis as the laptop's 64 GB. The buttons change the bits per weight of both.

Qwen3.6 read per token1.84 GB Qwen3.8 27B read per token16 GB Memory held against bytes read per token for the mixture of experts and the dense model on the laptop.

The MoE holds more (21.7 GB) and reads far less per token (about 1.84 GB, its 3 billion active weights plus the parts every token passes through). The dense 27B holds less and reads nearly all of it, about 16 GB, for every single token. Hold that ratio, about nine to one.

The bandwidth itself is a property of the board. LPDDR5X-8000 performs 8,000 million transfers per second, and the bus is 128 bits wide, 16 bytes per transfer: 8,000 million times 16 bytes is 128 GB/s. A wider bus means more pins on the chip, more memory chips and more power, which is why thin laptops stop at 128 bits; AMD's Strix Halo uses 256 bits, and Apple's M4 Max 512 bits for 546 GB/s1. A desktop graphics card has its own memory, VRAM, on a much wider bus: an RTX 5090 has 32 GB at 1,792 GB/s2. A model that fits in VRAM runs at VRAM speed; one that does not is split, part on the card and part on the CPU from system RAM, and the slow part sets the pace (ollama ps shows that split in its PROCESSOR column).

1Apple, New MacBook Pro features M4 family of chips, 30 October 2024: M4 Max with up to 546 GB/s of memory bandwidth. 2NVIDIA, Compare GeForce graphics cards: RTX 5090, 32 GB GDDR7, 1,792 GB/s; RTX 5080, 16 GB, 960 GB/s.

The laptop has no VRAM. Its integrated GPU, the iGPU, borrows system RAM (unified memory), so its memory is large and slow. That is a trade, and not a bad one: a 16 GB card cannot hold this 21.7 GB model at all, and the laptop holds it with room to spare. The chip has three kinds of processor around that memory. The CPU has 8 cores built to run one stream of instructions fast. The iGPU has 8 compute units of 64 simple arithmetic lanes each, 512 lanes doing the same instruction on different data, which is exactly what a dot product of 2,048 numbers wants. The NPU is a low-power matrix accelerator that neither Ollama nor llama.cpp uses, so it sits idle while the model runs.

Laptop iGPU Desktop RTX 5090 Cloud H100

Memory the GPU reads 64 GB shared 32 GB VRAM 80 GB HBM3

Bandwidth 128 GB/s 1,792 GB/s 3,350 GB/s

Holds Qwen3.6 35B-A3B (21.7 GB) yes yes yes

Holds a 60 GB model yes, slowly no, splits with the CPU yes

Compute units 8 CUs 170 SMs 132 SMs

Decode

We have every piece now, so let's follow the answer being written. The first token of the answer exists, and the model needs the second. Token 2 depends on token 1, which did not exist a moment ago, so there is no way around it: one full pass through all 40 layers, with a single column of numbers. That pass is a decoding step, and in it every active weight crosses the bus once to be used for one multiplication and one addition. This is the left end of the slider in the second figure, with real numbers.

Which gives us the simplest useful formula in this article. If every token has to read 1.84 GB, and the memory delivers at most 128 GB a second, the model cannot write more than 128 / 1.84, about 70 tokens a second.

laptop ceilingabout 70 tok/s laptop measured11.7 tok/s One second of memory bandwidth cut into tokens of 1.84 GB each: the ceiling on writing speed for each machine.

Each bar is one second of reading on one machine, all three on the same axis, and on the laptop you can see it cut into blocks of one token's 1.84 GB: about 70 of them fit. The bright ones are what the engine actually reaches, 11.7 tokens a second, measured. Switch to the dense model and the blocks grow ninefold. Real runs never touch the ceiling (on this laptop the memory delivers 30 to 40% of its theoretical bandwidth to this kind of access, and each of the 40 layers adds a small fixed cost for launching work on the GPU), but the ceiling is the right first guess, and it explains the ratio between models. The dense 27B reads about 16 GB per token, so its ceiling is 8 tokens a second, and it measured 2.6. Eight times the bytes, about four and a half times slower. That is the whole case for mixtures of experts on a machine like this one.

tok/s≤bandwidthbytes read per token\text{tok/s} \le \frac{\text{bandwidth}}{\text{bytes read per token}}tok/s≤bytes read per tokenbandwidth​

A third of the bandwidth is not what every machine gets. TokenWalk's timing model is fitted on published llama-bench runs, and in the 67 of them that measure a dense model writing, from an M1 to an MI300X, a single stream uses a median of 74% of the bandwidth on the spec sheet. The laptop's dense run uses 34% and the MoE 17%, the MoE lower because each of its layers reads so little that the fixed cost per layer weighs more. A Strix Halo, also an AMD iGPU, reaches 77 to 81% under llama.cpp, so being integrated does not explain the gap. How much of it is Ollama on Windows and how much is this chip I have not measured.

Share of the specified memory bandwidth one decoding stream uses, computed as tokens per second times bytes read per token over the spec sheet's GB/s. Grey: 67 published llama-bench generation runs of dense models (tg from an empty context) from TokenWalk's data/benchmarks.json, with bandwidths from data/machines.json and data/bench-hardware.json; bytes per token are the file size, and MoE rows are left out because theirs is not. Blue: this laptop's Ollama medians from the apu-llm benchmark, 11.7 and 2.6 tok/s, with bytes per token from the two GGUF files.

It also explains why quantization matters even when a model already fits. Fewer bits per weight means fewer bytes read per token, so halving the bytes roughly doubles the writing speed. Empty memory left over is worth nothing while the model runs; the bytes read are paid again on every token.

Meanwhile, a decode step does about two operations per weight, and a weight at 4.9 bits is about 0.61 bytes, so about 3.3 operations per byte read. The laptop's iGPU could do about 18 operations in the time it takes to fetch one byte. So it spends most of a decode step waiting. That waiting is what memory-bound means, and the ratio is the arithmetic intensity. The slider below is the number of tokens that share each read of the weights.

writing, 1 tokenabout 3 per byte balanceabout 18 per byte Operations per byte read against the laptop’s balance point, as the tokens sharing each read go from 1 to 512.

With one token the pin sits far on the memory side. Around eight tokens per read it crosses the balance point, and from there on the arithmetic is what limits. Writing is stuck at one token per read, because of the dependency between consecutive tokens. Reading a prompt is not.

Prefill

Now the minute of silence before the first word. The prompt is known in full before anything is generated, so all its tokens can go through the layers together, exactly as in training, with the causal mask keeping each one blind to the ones after it. This is prefill. The engine feeds the prompt in chunks of 512 tokens, so each weight read from memory serves 512 tokens, and the slider above says what happens then: far past the balance point, the chip is busy multiplying, and the reading hardly matters.

So prefill costs arithmetic. Qwen3.6 needs about 6 billion operations per token, 6,500 tokens of prompt are about 39 trillion operations, and the laptop's GPU manages around 0.6 to 0.75 trillion useful operations a second on this work. Divide, and you get a little over a minute: the minute of silence before the first word.

6,500 tokens on the laptopabout 70 s The wait before the first word as the prompt grows, on the laptop and on a desktop card.

The wait grows roughly in proportion to the prompt, and here a desktop card makes a much bigger difference than in decode, because the gap in compute units (8 against 170) is far larger than the gap in bandwidth. The two phases have the same arithmetic per token and opposite bottlenecks, and that is the most useful diagnosis I know for a local model.

Prefill

Decode

the whole prompt, in chunks of 512 Tokens per pass one

up to 512 tokens Each weight read serves one token

arithmetic (compute units) Limited by memory bandwidth

time to first token (TTFT) Felt as tokens per second after it

prompt eval rate Ollama reports it as eval rate

93 tok/s On the laptop, Qwen3.6 11.7 tok/s

If an answer is slow to start, it is prefill, and the levers are on the prompt side. Fewer and better passages shorten it directly, a summary can stand in for the whole history, and a fixed beginning lets the engine reuse earlier work (more on that below). If it writes slowly, it is decode, and the lever is bytes per token: a smaller format or a model with fewer active parameters reads fewer of them, and a machine with a wider bus reads them faster. The attention scores add a term that grows with the square of the prompt length, since every token is compared with every earlier one, but it stays below the rest of the pass until the context is several times the width of the model1.

1Kaplan et al., Scaling Laws for Neural Language Models, 2020, table 1 and section 2.1: the attention term is below the rest while the context is shorter than about twelve times the model width.

The KV cache

There is something I skipped in the decode step. The new token goes through attention, and attention compares its query with the keys of every earlier token, at every attention layer. Recomputing those keys each time would mean running the whole network over all 6,500 earlier tokens again, for every new token: about two million token-passes for a 300-token answer instead of 6,800.

The causal mask gives us the way out. The rows of earlier tokens never change, so their keys and values, once computed, stay correct forever. The engine keeps them, for every earlier token and every attention layer, and the new token's query is compared with them directly. That store is the KV cache. It is per layer (a new token at layer 39 reads what every earlier token had become by layer 39), and in Qwen3.6 only the ten full-attention layers keep one; the thirty DeltaNet layers keep their fixed state instead. The console's memory lens shows it filling one column per token.

Its size takes one line to count. Every attention layer stores one key and one value per token, each a set of KV heads times the head length, at so many bytes per number.

MKV=2×nlayers with KV×nkv heads×dhead×b×ntokensM_{KV} = 2 \times n_{\text{layers with KV}} \times n_{\text{kv heads}} \times d_{\text{head}} \times b \times n_{\text{tokens}}MKV​=2×nlayers with KV​×nkv heads​×dhead​×b×ntokens​

For Qwen3.6 in 16-bit numbers that is 2×10×2×256×2=20,4802 \times 10 \times 2 \times 256 \times 2 = 20{,}4802×10×2×256×2=20,480 bytes, 20 KB per token. Here is where the two choices we saw earlier pay off: grouped-query attention divided it by eight, and keeping it in only ten layers of forty divided it by four.

per token20 KB at 16,384 tokens335 MB The KV cache of Qwen3.6 against the context reserved, next to the same model if every layer kept one.

The dashed red line is about what the laptop has free once the model is loaded, and the lower bar crosses it at the right end of the slider. The memory for the cache is reserved when the model loads, for the whole context the request asks for (num_ctx), whether the conversation uses it or not. And this is where Ollama surprised me: it sizes the default from the memory it detects, and on this laptop it chose 262,144 tokens, which is 5.4 GB of cache reserved for a conversation of 7,000 tokens. For a model without the hybrid trick, the same reservation would be four times that. Setting OLLAMA_CONTEXT_LENGTH (or num_ctx per request) to what the application needs is the first setting I would change; the console with the 262,144-token reservation shows what it holds back.

Now we can do the whole memory budget of the laptop. Before any model was loaded, Windows with Chrome, Cursor, Claude Code and Anki open was already using 32 GB.

Qwen3.6 leaves freeabout 10 GB The laptop’s 64 GB filled by the system, a model’s weights, its KV cache and buffers.

With Qwen3.6 loaded and 16,384 tokens of context, about 10 GB stay free, which is about what I would want to keep for the system to breathe; the browser grows into whatever is there, and a model swapped to disk is unusable. With 64 GB the problem on this laptop is speed, never fitting. The cache can also be stored in fewer bits, separately from the weights (KV cache quantization, OLLAMA_KV_CACHE_TYPE=q8_0 for about half the memory). The difference is worth keeping straight: quantizing the weights rounds the model, the same for every request; quantizing the cache rounds the patient's history as the model has read it.

One more trick lives here. The grid of attention scores over NNN tokens has N2N^2N2 entries, and written out naively it sits in main memory between the step that makes it and the softmax that normalises it. FlashAttention never writes the grid out: it works through it in blocks that fit in the chip's own memory, with a running softmax1. The result is exactly the same numbers computed in a different order, with attention's memory growing linearly instead of quadratically. Ollama requires it (OLLAMA_FLASH_ATTENTION=1) for a quantized cache.

1Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.

Speculative decoding

Decode wastes the chip: one token per read, arithmetic units mostly idle. Prefill shows what the same chip does with many tokens per read. Speculative decoding borrows that. A small draft model that shares the big model's tokenizer (Qwen3 0.6B for a Qwen, say) writes a few tokens ahead, cheaply. The big model then checks all of them in one pass, which is a tiny prefill and costs about as much as one ordinary decode step, because the weights are read once for all of them. It keeps the draft's tokens from the left as long as they agree with what it would have produced, and at the first disagreement it puts in its own token and throws away the rest.

The output does not change. With sampling, a rejection rule makes the kept tokens follow the big model's distribution exactly1. Only the speed changes, and it depends on how often the draft is right, the acceptance rate α\alphaα.

1Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023, which proves the output distribution is preserved and derives the expected speedup.

tokens per round at α = 0.72.8 A round of speculative decoding with four draft tokens: the chance of reaching each slot before the first rejection.

With four draft tokens, the first token of a round is always produced (the big model writes it if the draft was wrong), the second needs one acceptance, the third two in a row, and so on; if all four are accepted, the same pass yields a fifth for free. At α=0.7\alpha = 0.7α=0.7 a round gives 2.8 tokens on average for one pass of the big model.

E[tokens per round]=1−αγ+11−αE[\text{tokens per round}] = \frac{1 - \alpha^{\gamma + 1}}{1 - \alpha}E[tokens per round]=1−α1−αγ+1​

Whether it pays depends on the model, and for our two models it comes out opposite, which is the part I found most interesting.

dense 27Bfaster MoEslower The time of one round with and without a draft model, for the dense 27B and for the mixture of experts.

For the dense 27B the check costs about one ordinary step (it reads 16 GB once whether it checks one token or five), and four steps of a 0.6B draft are cheap next to it, so a round is roughly twice as fast. For the MoE the same trick loses. Its ordinary step is cheap, since it reads 1.84 GB, and a check of five tokens is not: five tokens wake up to forty experts per layer instead of eight (the experts figure, with five tokens in the pass), so the check reads several times more, and the draft steps on top cost more than the tokens they save. My class notes had this as a hunch; the figure computes it from first-order costs, and measuring it is the experiment to run. The console runs both, the dense 27B with a draft and the MoE with the same draft. Qwen3.6's file even carries a built-in draft: its 41st block is a multi-token prediction layer trained to guess one token further ahead. Ollama does use it, by default with two drafts per round, and on this laptop it only pays when sampling is close to greedy: at temperature 0 it lifted writing from 13.2 to 15.6 tokens a second, at 0.7 it cost time instead (11.5 and 10.6 tokens a second in two runs, against 13.2 without it), because a cheaper guess gets rejected more often.

Prefix caching and the model lifecycle

A model remembers nothing between requests. The weights do not change, and the KV cache belongs to the request that built it. A chat feels continuous because the application sends the whole conversation again every time, from the system prompt down to the new question at the end. That whole text is the context window of that call, and every call pays prefill on all of it, unless the engine notices that it has seen the beginning before.

That is prefix caching. Call 4 of a conversation begins with the same tokens as call 3, and by the causal mask their keys and values are identical too, so an engine that kept them computes only the new tail. The match has to be exact from the very first token, and this is the part that bites in practice.

call 4, prompt appendedabout 350 tokens computed Four calls of one conversation: the tokens the engine reuses from the previous call and the ones it has to compute again.

Append the new question and call 4 computes a few hundred tokens. Change one word of the system prompt and everything after that word has different keys and values, because every later token attended to it, so call 4 recomputes all 7,600 tokens and pays the full minute again. Two habits follow: let the context grow by appending, with stable material first and the variable question last; and do not judge cost by counting tokens, since a cached 20,000-token prefix can be cheaper than a fresh 4,000-token one. APIs bill exactly this; Anthropic charges 1.25 times the input price to write a prefix into its five-minute cache and a tenth or less to read it back1. Ollama keeps the cache of the last request in the loaded model, which I discovered by accident when a benchmark reused the same prompt opening and read 2,900 tokens at 7,000 to 9,000 tokens a second.

1Anthropic, Prompt caching, pricing and the invalidation order tools, system, messages.

The cache lives in memory and dies with the model. llama.cpp's server can also save a slot's cache to disk (--slot-save-path) and load it back later. For the clinical copilot that changes the economics: the 6,500-token history costs about 70 seconds of prefill on this laptop and about 133 MB of cache, which an SSD reads back in a tenth of a second. Precompute each patient's cache once and restore it when the patient is opened, and the first token comes in about a second. For a hybrid model the saved state must include the DeltaNet layers' fixed state too, and I have not yet tested slot saving on this architecture.

The model itself can also be dropped from memory. Ollama is a background server (it listens on localhost:11434, and everything else, the ollama command included, is a client of it), and a model goes through five states with it.

1Pullonce, network time2Load55 s cold, disk to RAM3Serveprefill and decode4Idleholds memory5Unloadafter keep_alive

The setting that decides the last step, keep-alive, is five minutes by default, and it matters more than it looks for an interactive tool, because the first request after an unload pays the whole load again.

load on the laptop55 s Three questions on a timeline: with gaps shorter than keep_alive the model stays loaded; longer gaps make every question load it again.

Ask a question every three minutes and the model is always there. Ask one every ten minutes and every single question waits 55 seconds for the load before prefill even starts, as a cold start in the console shows. These are the settings the laptop needed, plus one that made the GPU usable at all:

OLLAMA_IGPU_ENABLE1without it Ollama ignores the 860M and runs on the CPU OLLAMA_CONTEXT_LENGTH16384default num_ctx; stops the 262,144-token reservation OLLAMA_FLASH_ATTENTION1blockwise attention, required for a quantized cache OLLAMA_KV_CACHE_TYPEq8_0cache at about half the memory OLLAMA_KEEP_ALIVE30mavoids reloading between bursts of use

Environment variables are read by the server when it starts, so they need a restart: on Windows, setx OLLAMA_CONTEXT_LENGTH 16384 and the rest, then quit Ollama from the tray and open it again; on Linux, sudo systemctl edit ollama with one Environment= line per variable. Setting them with export in a terminal does nothing to a server that is already running. Options sent in a request (num_ctx, keep_alive, temperature) override them for that call only, and every request reports the three numbers this article has been about:

PYTHON Copy

import requests r = requests.post("http://localhost:11434/api/generate", json={ "model": "qwen3.6:35b-a3b", "prompt": open("history.txt").read(), "stream": False, "options": {"num_ctx": 16384}, }).json() s = lambda k: r[k] / 1e9 # durations come in nanoseconds print("load", s("load_duration")) print("prefill tok/s", r["prompt_eval_count"] / s("prompt_eval_duration")) print("decode tok/s", r["eval_count"] / s("eval_duration"))

Ollama is not the only way to run a GGUF file, and the other two options are worth knowing.

Ollama LM Studio llama.cpp server

What it is a service with a CLI and an API a desktop app with a server the engine itself, compiled by you

Speculative decoding with a draft not exposed yes --model-draft

Saving the KV cache to disk no no --slot-save-path

Best for scripting, always-on service trying models, a GUI full control, research

Ollama hides the engine's options and manages files and memory for you; llama.cpp exposes everything and manages nothing. A project that needs speculative decoding or caches saved to disk moves to llama.cpp's server, and everything else is easier in Ollama.

Retrieval and the laptop's limits

Most long prompts in a real application come from retrieval-augmented generation: search your documents for passages relevant to the question and paste them into the prompt, so that the model answers from them and cites them. The search usually combines BM25, which scores passages by the exact words they share with the question (excellent with drug names like warfarin and amiodarone, blind to synonyms), with embedding search, which compares vectors from an embedding model and finds oral anticoagulant for warfarin (but can miss an exact code). Running both and merging the rankings is hybrid search. In the terms of this article, retrieval decides how much text enters prefill. At about 90 tokens a second on the laptop, ten passages of 500 tokens add almost a minute before the first word, so retrieving fewer, better passages is the cheapest speed-up a RAG system has.

The laptop runs a 35B model privately and for free per request, at about reading speed. It is the wrong machine for work that needs a lot of arithmetic for a long time. An evaluation over thousands of test cases is prefill-heavy, and so is a batch job over many documents; fine-tuning is training. That is exactly where 8 compute units hurt. A rented GPU does them in a fraction of the time (an RTX 5090 cost $0.69 an hour on RunPod's community cloud in October 20261, and it is what I used to fine-tune Stable Diffusion). The laptop keeps what must stay private, and an assistant that is always there.

1RunPod, Pricing, page updated 8 October 2026.

Back-of-the-envelope arithmetic

Most questions about whether a model will run, and how fast, can be answered before downloading anything, with the conversions we built along the way.

Size GB of a modelparameters in billions × bits per weight ÷ 835.5 × 4.9 ÷ 8 = 21.7 Bits per weight16 FP16, 8.5 Q8_0, ~4.9 Q4_K_MQ4_K_M is about 0.6 bytes Memory neededweights + KV cache + ~0.5 GB + 8 to 12 GB for the system Speed Decode ceilingbandwidth ÷ GB read per token128 ÷ 1.8 = 70 tok/s Decode, real30 to 50% of the ceiling on this laptop11.7 measured; ~75% in published dense runs Prefilluseful TFLOPS × 1e12 ÷ (2 × active parameters)0.6e12 ÷ 6e9 = 100 tok/s TTFTprompt tokens ÷ prefill tok/s6,500 ÷ 93 = 70 s Answer timeoutput tokens ÷ decode tok/s300 ÷ 11.7 = 26 s Cache KV per token2 × KV layers × KV heads × head size × bytes20 KB for Qwen3.6 Text 1,000 English words~1,300 tokensSpanish 1,300 to 1,600 Reading speed5 to 8 tok/sanything above it feels instant

The one I use most is the decode ceiling. A model that fits but writes below reading speed is a model you will stop using: on this laptop, 60 GB of dense weights fit in memory and would write about two tokens a second.

Short answers

I wrote this article to answer a list of questions I had after abandoning the class halfway. Here they are, each with its short answer.

Question Short answer

Do I need attention and Q, K, V to be an applied AI engineer? As mechanism, yes, because the KV cache and everything built on it follow from them. The derivations, no.

Is each layer a matrix times a vector? A layer is a few matrices; in decode each is multiplied by one vector, in prefill by a matrix of up to 512 columns.

Does decode only compute the new multiplications and some additions? It runs the new token's full pass through every layer, reading every active weight; it skips everything about earlier tokens.

Are intermediate results kept? Activations are overwritten layer by layer; only keys and values at attention layers are kept, in the KV cache.

Is the KV cache only for the first layer? It is per attention layer, all of them; in Qwen3.6 the 10 full-attention layers of 40.

Is a GGUF a big matrix of weights? A file of hundreds of named tensors behind a header whose metadata carries the tokenizer and the chat template.

Is quantization just cutting decimals? Rounding each weight to one of a few levels set by a shared scale per block; 4 bits are 16 levels.

Why do slides write ~1 byte and ~½ byte? Each block also stores its scale, so Q8_0 is 8.5 bits and Q4_K_M about 4.9.

Was it 27 GB or 17 GB? 27.3 billion parameters, 16.8 GB in Q4_K_M. Count and size are different numbers.

If the model fits, do fewer bytes still help? Yes, every token reads every active weight, so bytes per weight are paid per token.

Why tokens and not letters? Four times the positions: four times the decode passes.

How many tokens is a 300-word essay? About 400 to 500.

Could the last layer score only the top 10 tokens? Nobody knows which are top before scoring all 248,320, so engines compute them all.

What is temperature, internally? Every logit is divided by T before softmax.

Do probabilities not always sum to 1? Does sampling reorder them? They always sum to 1; top-p keeps the smallest top set reaching p and renormalises; nothing reorders.

Why can't I set Claude's temperature? The API rejects it since Opus 4.7, and on every model from version 2026-10-21; prompting replaces it.

Why 16 tok/s with the MoE? An early, unclean reading; the clean one is 11.7, against a ceiling near 70 set by 1.8 GB per token.

Is a dedicated NVIDIA card better because of CUDA? Because of its own fast memory (VRAM) and many compute units; CUDA is the software that uses them.

What are the CUs? Is the NPU a parallel processor? CUs are the iGPU's compute blocks (8 here); the NPU is a low-power matrix accelerator that Ollama does not use.

Does my laptop have a GPU? Yes, the integrated 860M, which also draws the screen.

Is decode limited by the CPU? By memory bandwidth, whichever processor runs it.

How is 128 GB/s a bottleneck on 64 GB? It reads the whole memory about twice a second; a token must read 1.8 to 16 GB of it.

What is a 128-bit bus, why not wider? 16 bytes per transfer, 8,000 million transfers a second; a wider one needs more pins on the chip and more power.

Is VRAM worse because it is smaller? Smaller and 7 to 25 times faster; a model that does not fit splits with the CPU and slows down.

Did I buy a bad computer? You bought capacity over speed: it holds models a 16 GB card cannot, at reading speed.

Is it always 8 experts? Does memory swap per token? 8 of 256 here, 2 of 8 in Mixtral; nothing swaps, every expert stays in memory.

Does every question resend everything? Yes, every time; prefix caching avoids recomputing the repeated start.

How much memory should stay free? 8 to 12 GB beyond the model; Windows and the apps here took 32 GB before it loaded.

What is fetch, decode, execute next to pull, load, serve? A CPU's instruction cycle, and a model's lifecycle in Ollama; unrelated, despite the shared word.

Why a small draft model if the big one does the same? It guesses cheaply; the big one checks several guesses in one read of its weights.

What does quantizing the KV cache mean? Rounding the stored context instead of the weights: half the cache memory at q8_0.

Trace of the request

Back to the question at the top, on the laptop, with everything in place.

1Ollama receives the messages over HTTPlocalhost:11434

1The chat template turns them into one ChatML string

1The tokenizer cuts it into about 6,550 token IDs

1Each ID becomes its row of the embedding table2,048 numbers

1Prefill runs 13 chunks of 512 through all 40 layersabout 70 s, the GPU's 8 CUs busy

1The ten attention layers store keys and values134 MB of KV cache

1The last column is scored against 248,320 tokensthe LM head

1The sampler draws the first token

1Each next token is one decoding stepreads ~1.9 GB, ~85 ms

1After about 300 steps the end-of-turn token stops the loop~26 s of writing

The minute of silence was prefill: 6,500 tokens times 6 billion operations each, on 8 compute units. The twelve tokens a second were decode: 1.84 GB read per token on a 128 GB/s bus, at about a third of its theoretical speed, with Ollama drafting two tokens per step through the model's own prediction head, which at this temperature costs about as much as it saves (13.2 tokens a second with drafting off, 15.6 with it on at temperature 0). Almost every part of this pipeline existed before 2017, attention included. What the transformer changed was the order in which work can happen: training reads every position at once, which is why it scaled. Generation is the one place where that parallelism ends, one token at a time through memory, and every trick in the second half of this article is a way to make that one sequential step read fewer bytes.

11.7 tok/sQwen3.6 35B-A3B writing on the Radeon 860M, against a bandwidth ceiling near 70the measurements of the previous article 93 tok/sthe same model reading a 2,900-token promptsame run 2.6 tok/sthe dense Qwen3.8 27B writing, which reads about 16 GB per tokensame run 20 KBKV cache per token of Qwen3.6 in f16, from 10 attention layers of 40computed from the GGUF header 4.89 bitsaverage bits per weight of the 21.7 GB filesummed from its 753 tensors 2.8tokens per round of speculative decoding at α = 0.7 and four draft tokensLeviathan et al., 2023