ML//training//contrastive learning

Contrastive learning is a training method that teaches a model from pairs and triplets (a query, a positive match and a negative match), bringing positive pairs closer in vector space and pushing negative pairs apart, so that the model's vectors capture semantic similarity for search and retrieval.

Contrastive learning is a training method that teaches a model from pairs and triplets (a query, a positive match and a negative match), bringing positive pairs closer in vector space and pushing negative pairs apart, so that the model's vectors capture semantic similarity for search and retrieval.

For RAG, the query Capital of France? with Paris is the capital should reach a high cosine similarity, and the same query with Tortilla recipe a low one.

It is the loss that makes sentence transformers work: it turns BERT's [CLS] vector from one that predicts masked tokens into one that captures semantic similarity.

Without contrastive fine-tuning, base BERT does semantic search poorly, because its [CLS] vector was never optimized for similarity.

It is also used in CLIP (text-image pairs), in self-supervised learning and in representation learning broadly.