ML//transformer//tokenizer//SentencePiece

SentencePiece is a language-agnostic tokenization library (Google, 2018) that treats text as a raw byte or Unicode stream, not pre-tokenized words. It learns a subword vocabulary directly from raw text, without needing language-specific word boundary rules.

SentencePiece is a language-agnostic tokenization library (Google, 2018) that treats text as a raw byte or Unicode stream, not pre-tokenized words. It learns a subword vocabulary directly from raw text, without needing language-specific word boundary rules.

It implements both the BPE and Unigram algorithms, and most modern models use its BPE implementation. Its key difference from earlier BPE tools (like GPT-2's tokenizer) is that SentencePiece does not require pre-tokenization (splitting on whitespace first), so it handles languages without spaces (Chinese, Japanese, Thai) natively.

It is used by LLaMA, T5, and most multilingual models. The whitespace-agnostic design is why these models handle non-English text better than GPT-2/3 (which assumed English-like word boundaries)

The SentencePiece model (the .model file) contains the learned vocabulary and merge rules. It's fully self-contained: no external dictionaries or language-specific logic. This makes it easy to share tokenizers across different codebases and frameworks.

SentencePiece can operate at the byte level (byte-fallback mode), like byte-level models, which covers any input completely. Unknown characters are decomposed into UTF-8 bytes, each mapped to a special byte token, and that eliminates the unknown token problem entirely.

The vocabulary size is a trade-off. A larger vocab (32K-100K) means more concepts are single tokens (efficient), but the embedding matrix grows proportionally (more parameters). Smaller vocab means more tokens per concept (slower generation) but a smaller model. Most LLMs settle on 32K-128K tokens.