ML//model//BERT

BERT (Bidirectional Encoder Representations from Transformers, Google, 2018) is an encoder-only transformer that sees the full input in both directions, with no causal masking, unlike GPT's left-to-right reading. It was trained with masked language modeling (predicting hidden tokens) and NSP (predicting whether two sentences are consecutive, later found unhelpful).

BERT (Bidirectional Encoder Representations from Transformers, Google, 2018) is an encoder-only transformer that sees the full input in both directions, with no causal masking, unlike GPT's left-to-right reading. It was trained with masked language modeling (predicting hidden tokens) and NSP (predicting whether two sentences are consecutive, later found unhelpful).

Without causal masking every position is enriched better: early words get fully disambiguated by later context. The price is that BERT cannot generate text.

Its output is one vector per token, with the [CLS] token used for sentence-level tasks; downstream layers convert the vectors into task outputs.

It dominated NLP benchmarks for two years and proved that pre-training followed by fine-tuning dominates NLP. Fine-tuned with contrastive learning, it became the sentence transformers used for RAG.