ML//RAG

Retrieval-augmented generation: before answering, retrieve relevant documents and stuff them into the context window alongside the question.

Retrieval-augmented generation: before answering, retrieve relevant documents and stuff them into the context window alongside the question.

Sources: web search, databases, PDFs (retrieved from a vector database using cosine similarity)

Powered by sentence transformers: BERT fine-tuned with contrastive learning so that [CLS] captures semantic similarity instead of just predicting masks.

GPT doesn't use [CLS]: it takes the last token as the embedding (optimized via causal masking to carry full meaning)

Training is expensive and slow. RAG is cheap, instant, pure context window manipulation.

The dark side: if the retrieved documents are garbage, the model will confidently synthesize garbage. RAG is only as smart as its search engine.