ML//transformer//tokenizer//BPE

BPE (Byte Pair Encoding) is a tokenization algorithm that builds a subword vocabulary by starting with individual characters and iteratively merging the most frequent adjacent pair until the target size is reached, and it is the most common way to build the tokenizer of a language model. It is a simple greedy algorithm and a surprisingly effective compression.

BPE (Byte Pair Encoding) is a tokenization algorithm that builds a subword vocabulary by starting with individual characters and iteratively merging the most frequent adjacent pair until the target size is reached, and it is the most common way to build the tokenizer of a language model. It is a simple greedy algorithm and a surprisingly effective compression.

The GPT family uses BPE, and its vocabulary is a frozen artifact of the training data.

Words like running, runner and runs share subword tokens, so the model gets morphological hints for free, unlike with character-level tokenization.

The current trend is BPE with large vocabularies (around 100K tokens), which balances sequence length against embedding sparsity.