ML//inference//local inference//GGUF

GGUF is the file format of the llama.cpp ecosystem for storing a language model in one self-contained file: its weights, usually quantized, its tokenizer, its chat template and the metadata an engine needs to run it. llama.cpp, Ollama and LM Studio all read it, so the same file can be copied or hard-linked between them and runs on identical weights.

GGUF is the file format of the llama.cpp ecosystem for storing a language model in one self-contained file: its weights, usually quantized, its tokenizer, its chat template and the metadata an engine needs to run it. llama.cpp, Ollama and LM Studio all read it, so the same file can be copied or hard-linked between them and runs on identical weights.

The file has four parts in order: a header (the four bytes GGUF, the version, the number of tensors and of metadata keys), a metadata dictionary, a table that describes each tensor (name, shape, number format, offset) and the tensor data, aligned, which is almost the whole size. The metadata of Qwen3.6 35B-A3B, read from the file Ollama downloads, gives general.architecture = qwen35moe, 41 blocks (40 layers plus one for multi-token prediction), a width of 2,048, 16 query heads and 2 key-value heads of 256, one full-attention layer in four, 256 experts with 8 used per token and a context of 262,144, and it carries the 248,320 tokenizer entries and 247,587 BPE merges.

Nothing in a GGUF is code: the engine brings the arithmetic, the file brings the numbers and their shapes. Reading its header with a few lines of Python is the quickest way to learn a model's real architecture, including details its model card leaves out.

Tensor names spell out the architecture. blk.3.attn_q.weight is the query projection of layer 3, blk.0.ffn_up_exps.weight packs the 256 experts of layer 0 into one tensor of shape [2048, 512, 256], and output.weight is the output head.

The quantization label in a file name is an average. Q4_K_M stores most matrices in Q4_K at 4.5 bits per weight and the sensitive ones in Q6_K at 6.56; that file averages 4.89 bits, 21.7 GB for 35.5 billion parameters (quantization).

Ollama stores each GGUF as a content-addressed blob under ~/.ollama/models/blobs, and the largest blob of a model is the file itself.