ML//model//BERT//next sentence prediction

Next sentence prediction is a pre-training objective, BERT's second alongside MLM, in which the model is given a sentence pair (A, B) and predicts whether B actually follows A in the original text. It is a binary classification (is next or not next) over 50% real pairs and 50% random ones.

Next sentence prediction is a pre-training objective, BERT's second alongside MLM, in which the model is given a sentence pair (A, B) and predicts whether B actually follows A in the original text. It is a binary classification (is next or not next) over 50% real pairs and 50% random ones.

It is the pre-training objective that got fired. BERT shipped with it, and it was later found to contribute little: models such as RoBERTa dropped NSP entirely and performed better. The intuition was that it would teach sentence-level reasoning, but MLM alone turned out to capture this sufficiently.