ML//transformer//tokenizer//special tokens

Special tokens are tokens that carry structural meaning rather than linguistic meaning. Four of them recur across models:

Special tokens are tokens that carry structural meaning rather than linguistic meaning. Four of them recur across models:

[CLS]BERT's aggregation token. It sits at position 0, learns to capture global sentence meaning during training and is used for classification tasks via downstream layers.

[MASK]The placeholder for masked language modeling. The model predicts what goes there using bidirectional context.

[BOS]Beginning of sequence. In cross-attention translation, the query from [BOS] is the cold start: it has to attend to the source language before anything has been generated.

[EOS]End of sequence. When the model generates this token, inference stops.

GPT does not use [CLS]. It simply takes the last token's representation, already optimized to carry the full sequence meaning through causal masking.