ML//model//GPT//causal language modeling

Causal language modeling is GPT's training objective: predicting the next token using only the tokens to its left, with no [MASK] and no bidirectional context. It is the opposite of MLM (BERT), which predicts hidden tokens from both sides, while CLM predicts the future from the past.

Causal language modeling is GPT's training objective: predicting the next token using only the tokens to its left, with no [MASK] and no bidirectional context. It is the opposite of MLM (BERT), which predicts hidden tokens from both sides, while CLM predicts the future from the past.

The causal mask enforces this during training: each position can only attend to previous positions.

Training produces T−1T-1T−1 examples from a sequence of TTT tokens, because every position is a prediction target.

The objective stayed the same from GPT-1 through GPT-4: the data and the model were scaled, and the basic task did not change.