hardware//RAM//unified memory

Unified memory is a design in which the CPU and the GPU (and often an NPU) of one chip share a single pool of system RAM instead of the GPU having memory of its own, and it matters for local inference because it lets an integrated GPU hold models far larger than a consumer graphics card can. Apple Silicon, AMD's APUs (Ryzen AI, Strix Halo) and phone SoCs are built this way.

Unified memory is a design in which the CPU and the GPU (and often an NPU) of one chip share a single pool of system RAM instead of the GPU having memory of its own, and it matters for local inference because it lets an integrated GPU hold models far larger than a consumer graphics card can. Apple Silicon, AMD's APUs (Ryzen AI, Strix Halo) and phone SoCs are built this way.

The pool is large and the road to it is narrow. A ThinkPad with 64 GB of LPDDR5X-8000 on a 128-bit bus gives its Radeon 860M access to most of the 64 GB (Ollama counted 47.6 GiB usable) at 8,000 million transfers a second times 16 bytes, 128 GB/s, shared with the CPU. A desktop card has 16 to 32 GB of VRAM at 1,000 to 1,800 GB/s.

Unified memory trades bandwidth for capacity. It holds a 21.7 GB model that a 16 GB card cannot hold at all and writes with it at reading speed, while a dense model of the same size, which reads 16 GB per token, crawls at two or three tokens a second.

The bandwidth is fixed by the board: transfer rate, bus width and channels. Thin laptops stop at 128 bits; Strix Halo uses 256 bits (256 GB/s) and Apple's M4 Max 512 bits (546 GB/s), which is why those machines run large models much faster.

On Windows the firmware sets aside a small fixed slice for the GPU (512 MB by default on that laptop) and the rest is borrowed on demand as shared memory; reserving a larger slice takes a reboot and takes the memory from every other program.

Because the CPU and the GPU read the same memory, they write tokens at nearly the same speed. The GPU's advantage shows in prefill, where arithmetic is what counts.