ML//model//BERT//masked language modeling

Masked language modeling (MLM) is a pretraining objective in which 15% of the input tokens are randomly replaced with [MASK] and the model is asked to predict them from the context on both sides, and it is the objective BERT is trained with. The output is one prediction per masked position, made with the full context (past and future tokens) to disambiguate.

Masked language modeling (MLM) is a pretraining objective in which 15% of the input tokens are randomly replaced with [MASK] and the model is asked to predict them from the context on both sides, and it is the objective BERT is trained with. The output is one prediction per masked position, made with the full context (past and future tokens) to disambiguate.

The objective is deceptively simple and forces genuine bidirectional understanding: the model cannot just copy from left to right, and has to reason about the masked position from the surrounding words. It differs from causal language modeling (GPT), which predicts the next token left to right, where MLM predicts hidden tokens in any position.