ML//transformer//tokenizer//byte-level model

A byte-level model is a language model that tokenizes its input at the byte level (256 possible tokens) instead of using subwords, so that no text is ever out of its vocabulary. ByT5 and MegaByte (Meta, 2023) are real research models built this way.

A byte-level model is a language model that tokenizes its input at the byte level (256 possible tokens) instead of using subwords, so that no text is ever out of its vocabulary. ByT5 and MegaByte (Meta, 2023) are real research models built this way.

The sequence is about 4 times longer than in characters, which makes it about 16 times more expensive than characters and about 100 times more expensive than BPE, because attention is O(n2)O(n^2)O(n2).

The bet is that at the byte level the model learns an optimal compression without tokenizer biases: perfect handling of typos, new languages, invented words and code, and never an out-of-vocabulary token.

MegaByte reduces the quadratic cost with local attention over byte patches and global attention between patches.

Meta's Byte Latent Transformer (BLT, December 2024) attacks that cost by grouping bytes into patches of variable length. A small model estimates how predictable the next byte is, and a new patch starts where the text becomes hard to predict, so easy stretches travel in long patches and the compute goes where the information is. The paper reports matching Llama 3 at the 8-billion-parameter scale for the same training compute, with up to half the inference compute and better tolerance of noisy input; it works on bytes of eight bits, never on single bits.

The cost of plain byte models kept them in research for years. BLT is the first evidence that a model without a tokenizer can keep pace with tokenized ones at scale, and production models are still tokenized.

It sits below the token as a Large Concept Model sits above it: both change the unit the model predicts, and each moves a cost somewhere else.