ML//transformer//tokenizer

A tokenizer is the algorithm that breaks a model's input text into subword tokens and assigns each one an ID, the first step in turning text into vectors, so that *unbelievable* becomes *un*, *believ* and *able*. The pieces of the input are the tokens, and they can also be chunks of an image or of sound.

A tokenizer is the algorithm that breaks a model's input text into subword tokens and assigns each one an ID, the first step in turning text into vectors, so that unbelievable becomes un, believ and able. The pieces of the input are the tokens, and they can also be chunks of an image or of sound.

Two separate steps turn text into vectors. The tokenizer (an algorithm, not a neural network) splits the text and assigns token IDs; the embedding matrix then converts the IDs into vectors:

1The word gato2Token ID 48213A vector of 512 dimensions

Vocabulary size is a tradeoff: a small vocabulary (8K) gives more tokens per sentence, a large one (100K) is sparser but gives shorter sequences.

BPE is the most common algorithm; WordPiece (BERT) and SentencePiece are the alternatives.

A token is not a word. Common words are one token, rare words and part numbers break into several (a tag like P-101 can take four), and spaces and punctuation count too. Prices, context limits and generation speed are all counted in tokens, so the same document costs more in a language or a format the tokenizer saw little of. A byte-level model drops the tokenizer altogether and pays in sequence length.

The tokenizer is almost never touched in fine-tuning, because changing it means changing the embedding matrix, which is basically starting over.

?Why not use characters instead of subwords?

The word transformer is 2 or 3 BPE tokens but 11 characters, so characters fit about 3× less text in the context window, and the model must learn morphology from scratch (running, runner and runs lose their shared subwords). Characters do give perfect typo handling and cope with new languages and code, but at current scale the trade-off favors BPE.