ML//Transformer//tokenizer//special tokens

Tokens with structural meaning, not linguistic meaning:

Tokens with structural meaning, not linguistic meaning:

[CLS]: BERT's aggregation token. Sits at position 0, learns to capture global sentence meaning during training. Used for classification tasks via downstream layers

[MASK]: placeholder for masked language modeling. The model predicts what goes here using bidirectional context.

[BOS]: beginning of sequence. In cross-attention translation, the Q from [BOS] is the "cold start": how do you attend to the source language when you haven't generated anything yet?

[EOS]: end of sequence. When the model generates this token, inference stops.

GPT doesn't use [CLS]. It simply takes the last token's representation (already optimized to carry full sequence meaning via causal masking)