ML//transformer//tokenizer//vocabulary

The vocabulary is the finite set of tokens a model knows: every possible subword unit it can produce or consume. It determines the shape of the embedding matrix, with one row per vocabulary entry and one column per embedding dimension.

The vocabulary is the finite set of tokens a model knows: every possible subword unit it can produce or consume. It determines the shape of the embedding matrix, with one row per vocabulary entry and one column per embedding dimension.

Its size is a tradeoff. A small vocabulary (8K) gives longer sequences and spends more context per sentence; a large one (100K) gives shorter sequences and sparser embeddings.

BPE builds the vocabulary greedily from the character level up, merging the most frequent pairs until it reaches the target size.

The vocabulary is a frozen artifact of the training data: it is fixed at pre-training time and never touched during fine-tuning.