ML//red teaming

Red teaming is an evaluation practice in which people (or AIs) are hired to break an LLM on purpose, stress-testing its behavior so that the failures they find can be fixed. It is the empirical side of alignment: it finds the failures that theoretical frameworks miss. Cybersecurity evaluation turns selected offensive and defensive failure modes of this kind into repeatable tasks, and a finding only becomes operationally meaningful once exposure, permissions and containment are weighed (deployment risk).

Red teaming is an evaluation practice in which people (or AIs) are hired to break an LLM on purpose, stress-testing its behavior so that the failures they find can be fixed. It is the empirical side of alignment: it finds the failures that theoretical frameworks miss. Cybersecurity evaluation turns selected offensive and defensive failure modes of this kind into repeatable tasks, and a finding only becomes operationally meaningful once exposure, permissions and containment are weighed (deployment risk).

In a model's release cycle it runs during and after post-training, often for months before launch, and it overlaps with the last training rounds instead of following them: what the red team breaks becomes the next round's data.

Its findings become fuel for training at three depths: SFT as a quick patch, Constitutional AI feeding DPO as the deep fix, and system prompt guardrails as a bandaid. It is an arms race with no finish line, because every patch creates new edges and every new capability creates new attack surfaces.

Red teaming sits among other ways of evaluating and steering a model:

BenchmarksExams on LLM intelligence, usually scored

Safety trainingLike benchmarks, but for the 500 known danger categories instead of intelligence

GuardrailsPure system prompt injection with no training, faster than SFT or DPO but the most brittle

Few-shot learningNo training at all, the weights stay frozen and examples in the prompt narrow down the latent space