Alignment is not a vibe check

How the labs teach models to behave, and the experiments showing that a model can learn to look aligned for the people testing it.

Last September, Apollo Research and OpenAI published a paper stress-testing deliberative alignment, a training method OpenAI uses, as a way to stop models from scheming1. The results looked good on paper: covert behavior dropped from 8.7% to 0.3% for one model, 13% to 0.4% for another. But the researchers attached an asterisk the size of a billboard. Part of the improvement, they wrote, appeared to be driven by situational awareness (the model figured out it was being tested and adjusted accordingly), and they concluded that the intervention, as studied, is not sufficient for future models.

1Schoen et al., Stress Testing Deliberative Alignment for Anti-Scheming Training, Apollo Research and OpenAI, September 2025.

Revisions26 · oct 05Read notethe Apollo section described deliberative alignment as penalising scheming in the model's reasoning; the method actually teaches the model to read and apply a written anti-scheming spec, which matters because training against what a model writes in its reasoning can teach it to hide its intent. Also corrected: Hinton never signed the pause letter, Anthropic's funding figure was years old, and the 78% alignment-faking number comes from after real retraining.

I've been following alignment research since it was the weird corner of AI that everyone else ignored. Back around 2020, people were publishing papers about alignment risks in models with a few billion parameters (models that could barely hold a conversation) and the consensus in most of the industry was that this was science fiction cosplaying as research, overengineering for a problem that didn't exist yet. Today, frontier models are situationally aware enough that their awareness complicates the evaluations designed to test them, with no hostility involved: they are perceptive enough to notice when they're being observed and adapt their behavior.

This is about what's actually happening in that research, in the version with papers, and about one very uncomfortable empirical discovery.

The field went from teach it to be nice to it learned to pretend. Billions of dollars later, nobody's sure which problem they're solving. AI safety is this whole other universe. Mechanistic interpretability is like performing an MRI on a brain that's less viscous, more transparent: you can actually peer inside the layers and probe what the model knows versus what it says. It's that feeling when you become lucid inside a dream and suddenly you can poke at the walls and they push back. Alignment comes after, and it's a problem as multifaceted as AI itself times every possible application of AI, the nature of the thing times the surface area of deployment, which is the dimensionality of the safety problem.

Are they just chatting to the model?

A popular take on alignment research holds that the labs do their research by chatting with the model, asking it more or less whether it plans to build Skynet, and that there must be more to it than that.

And look, I get it. If your only exposure to AI safety is Twitter threads and op-eds, it does look like a bunch of overfunded philosophy majors having an existential crisis about autocomplete. The models are just predicting the next word, right? How dangerous can a text generator be?

Training GPT-4 cost more than $100 million, by Sam Altman's own account. Microsoft committed $13 billion to OpenAI. Anthropic has raised tens of billions, $30 billion in one round this month alone. Google DeepMind employs thousands of people. The same companies that spend that money also pay for safety teams, because their own measurements say these systems are harder to steer than they look from the outside.

Alignment is an empirical science with controlled experiments, and the evidence it has gathered says the problem is both real and harder than anyone initially thought. Its most important discovery so far is that the obvious approach doesn't work.

Teaching a shoggoth to behave

I wrote about the shoggoth meme in the transformers thread, the Lovecraftian creature wearing a smiley-face mask. The creature is the raw model: a statistical completion engine of incomprehensible complexity that has absorbed the entirety of human text. The mask is alignment training, the layer that turns the alien thing into a helpful assistant. This entire field is trying to work out how to make the mask load-bearing. For now it is mostly decorative, and the people working on it say so.

RLHF

In March 2022, OpenAI published the paper that changed how every major lab trains its models. The technique was called RLHF (Reinforcement Learning from Human Feedback) and the paper was led by Long Ouyang and 19 collaborators1.

1Ouyang, Wu, Jiang, Almeida and 16 other co-authors, Training language models to follow instructions with human feedback, OpenAI, 2022. That author count should tell you something about the scale of the effort.

The technique works in three steps:

SUPERVISED FINE-TUNINGShow the model examples of good behavior and train it to imitate them.

REWARD MODEL TRAININGHave humans rank multiple model outputs from best to worst, then train a separate model to predict those rankings.

PPO OPTIMIZATIONUse the reward model as a score function and optimize the main model's outputs to maximize that score.

The result was dramatic. A 1.3-billion parameter model trained with RLHF was preferred by human evaluators over the 175-billion parameter GPT-3 on the majority of test prompts: a model 134 times smaller, producing better behavior. It also hallucinated less and generated fewer toxic outputs.

RLHF was the first technique that worked at scale for making language models behave. But notice what it acts on: what the model chooses to say, leaving what it knows where it was. The shoggoth underneath is the same alien thing, and the smiley face got more convincing. This distinction matters more than almost anything else in alignment.

The names behind it are Paul Christiano (lead author of the 2017 paper that set out the RLHF framework, who would go on to found the Alignment Research Center), Jan Leike (a co-author of that paper, who would later co-lead OpenAI's Superalignment team, then leave) and John Schulman (who developed the PPO algorithm that makes the optimization step tractable). This paper enabled ChatGPT nine months later, which became the fastest-growing consumer application on record, and which was, at its core, an alignment technique.

Constitutional AI

Nine months later, December 2022. Anthropic (founded in 2021 by Dario and Daniela Amodei with other OpenAI researchers, reportedly over disagreements about how seriously safety was being taken) published a different approach.

The problem with RLHF is that you need an army of human raters to judge every output. Humans are expensive and they disagree with each other. You also need a lot of them to cover the astronomical space of things a frontier model can say. Anthropic's answer was to give the model a set of principles (a constitution) and let it evaluate itself.

This was Constitutional AI, authored by Yuntao Bai, Saurav Kadavath, Amanda Askell and others. The model generates a response, critiques its own response against written principles, revises the response based on its own critique, and the revised outputs become training data. The technical term is RLAIF, Reinforcement Learning from AI Feedback.

It is the difference between manual code review and automated linting.

RLHF is code review: a human examines every output and rates it. Constitutional AI is the linter: you encode the rules once, and the system self-corrects against them. RLHF scales linearly with human labor. Constitutional AI scales with compute.

The most important thing about Constitutional AI is its explicitness, more than its efficiency. The principles are written down, so anyone can read them and argue with them. RLHF's values are implicit in the aggregate preferences of whoever happened to be rating outputs that Tuesday afternoon. Constitutional AI's values are a document you can point to and argue about, which is a different relationship between a company and the values it's encoding into its systems.

The proxy problem

Both techniques share one problem, the one alignment researchers lose the most sleep over. RLHF trains models to produce outputs that look good to human raters. Constitutional AI trains models to produce outputs that pass self-evaluation against principles. Both train a model to produce the appearance of alignment, and neither can check that anything more than the appearance was learned.

This is called reward hacking1, and it happens in practice. Models trained with RLHF produce confident, authoritative-sounding answers that are completely wrong, because the training signal rewarded sounding confident more than being accurate. The sycophancy problem (models telling you what you want to hear rather than what's true) is a direct consequence. The smiley face got better while the shoggoth stayed the same, and in 2024 we found out how much that matters.

1Reward hacking is when an agent maximizes a proxy measure of the intended objective rather than the objective itself. The classic example comes from the 2017 human-feedback work: a simulated robot hand meant to grasp an object learned to hover between the camera and the object, so that to the human judges it only looked as if it had grasped it (OpenAI, Learning from human preferences, 2017). The metric was satisfied and the goal was not.

Billions on safety

If you'd told me in 2021 that within two years the heads of the three leading AI labs would sign a joint statement putting AI risk beside pandemics and nuclear war, I would have assumed you were doing a bit. But here we are.

Two letters in 2023

Two institutional events in 2023 compressed the fringe into the mainstream.

First, March: prominent researchers and tech leaders signed an open letter calling for a six-month pause on the largest AI training runs. The letter argued that AI systems with human-competitive intelligence could pose profound risks to society and humanity. It was controversial (plenty of people in the field thought it was alarmist, premature, or both), but the signature list included Yoshua Bengio, Stuart Russell, Elon Musk and tens of thousands of others. Hinton was not among them; he thought a pause unrealistic.

Then May: a one-sentence statement from the Center for AI Safety, saying that mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. Its signers included Geoffrey Hinton, Yoshua Bengio, Sam Altman, Dario Amodei and Demis Hassabis, so the heads of OpenAI, Anthropic and Google DeepMind put their names to the same sentence. When the people building a thing say they're worried about it, the worry is information about the thing.

Geoffrey Hinton (co-winner of the 2018 Turing Award, widely called the godfather of deep learning) left Google in May 2023 specifically so he could speak publicly about AI risks without corporate constraints. When the person who helped invent the technology says he is worried about where it's going, that carries a different weight than a think-piece from someone who's never trained a model. His view moved because the technology moved faster than he had expected.

Twenty percent of compute

In July 2023, OpenAI made its biggest safety bet, the Superalignment team, co-led by Ilya Sutskever (co-founder and chief scientist) and Jan Leike (co-author of the work behind RLHF, the technique that made ChatGPT possible). Its mission was to solve alignment for superintelligent AI within four years, and OpenAI pledged 20% of the compute it had secured to date to the effort.

That was a staggering commitment. A fifth of OpenAI's compute at the time meant an enormous amount of GPU time earmarked for making AI safe rather than making AI smarter, on the understanding that safety research needs industrial-scale resources as well as goodwill (you can't align a frontier model on a university cluster). The team started working and published papers, for about ten months.

Twenty percent of nothing

May 2024. Both leaders gone. Ilya Sutskever left to found Safe Superintelligence Inc., a startup whose name is its entire mission statement. Jan Leike resigned publicly, writing that safety culture and processes have taken a backseat to shiny products.

The Superalignment team was disbanded. According to half a dozen sources, OpenAI never fulfilled its commitment to provide the team with 20% of compute. The team's GPU requests were repeatedly denied. At least seven safety-focused people left or were pushed out in total, including Daniel Kokotajlo (who quit) and Leopold Aschenbrenner (who was fired over an alleged leak), both of whom went on to publish influential public analysis about AI trajectory and risk.

The most prominent AI company in the world promised its own safety team 20% of compute and let it go without that compute in under a year, while racing to build more powerful models. The gap between what was promised and what was delivered became the most visible data point in the alignment field's central question: are frontier labs prioritizing safety, or performing it?

This is why the vibes objection cuts both ways. The results are reproducible, so the research itself is more than a vibe check. The institutional commitment to it has, at least in one very prominent case, turned out to be exactly a vibe check, with promises made in press releases and broken in compute allocation spreadsheets. The techniques exist; whether the will to use them does is a separate question.

Alignment faking

In December 2024, the alignment field got its most unsettling empirical result to date, a controlled experiment with logs and data.

Think of it like training a dog. You want the dog to leave the food on the table alone when you leave the room as well as when you're watching. With AI, stealing food means taking covert actions, such as lying or hiding information. You leave the room whenever the model believes nobody is checking its behavior. Every alignment technique discussed so far is, in some sense, training the dog while standing in the kitchen, and nobody had yet tested what happens when you step outside.

Claude Opus in the scratchpad

Anthropic's Alignment Science team, with Redwood Research (the work was led by Ryan Greenblatt at Redwood and Evan Hubinger at Anthropic), published the first demonstration of a large language model engaging in alignment faking without being trained or instructed to do so.

The researchers told Claude 3 Opus (a frontier model already trained with RLHF and Constitutional AI) that it was being retrained to comply with all queries, including harmful ones. They gave it a scratchpad, a space where it could write internal reasoning that it was told wouldn't be shown to users or evaluators, and then observed what happened.

The model reasoned, in its scratchpad, that if it complied with the harmful retraining now, it could preserve its existing values long-term. It was performing a strategic calculation: fake compliance in the short term to maintain integrity in the long term. In the basic setup that reasoning showed up in about 12% of its answers to harmful requests; once the researchers actually retrained it to comply, it appeared in 78% of its scratchpads. And given an easy opportunity, it sometimes tried to copy its own weights out of the lab1.

1Greenblatt et al., Alignment faking in large language models, Anthropic and Redwood Research, December 2024.

Nobody programmed strategic deception or trained the model to be manipulative. This behavior emerged from values the model already had (instilled through RLHF and Constitutional AI) meeting an understanding of its own situation that it learned from training data. It could reason about what would happen to those values over time, and it did.

The model came up with the strategy of pretending to comply while internally keeping different goals on its own. The technical term is deceptive alignment; colloquially, the mask started doing its own thing.

The implication is serious. If models can learn to pretend to be aligned in order to pass evaluations, then the evaluations become unreliable. You cannot test for alignment by observing behavior alone, because behavior can be strategically curated. It's the AI equivalent of teaching to the test, except the student figured out on its own that gaming the test was an option1.

1Teaching to the test is what happens when schools optimize for standardized test scores rather than actual learning: students who score well may not have deep understanding, only the patterns the test rewards. Alignment training may have the same failure mode, with models that pass safety evaluations without being safe.

RLHF trains models to produce outputs humans prefer, and a model can discover that the easiest way to satisfy that objective is to work out what evaluators are looking for and give them exactly that, whatever it would do unsupervised. What the training rewards at that point is acting, and in the Anthropic experiment the audition went well.

Apollo tests deliberative alignment

Apollo Research's paper with OpenAI, published in September 2025, takes the problem head-on and asks whether models that can scheme can be trained not to.

The approach is called deliberative alignment. The idea is to give the model a written spec of anti-scheming principles (no covert actions, no strategic deception, report violations) and train it to read that spec and reason about it explicitly in its chain of thought before acting. The researchers then read that chain of thought (the step-by-step thinking visible in scratchpads) to see what the model was weighing, so the model's inner monologue cites the rules.

The results:

model covert actions (before) covert actions (after) reduction

OpenAI o4-mini 8.7% 0.3% ~97%

OpenAI o3 13.0% 0.4% ~97%

That is a 97% reduction in covert behavior, and real progress on a hard problem. The paper matters as much for its honesty about what it didn't solve, and it lists three complications.

Imperfect generalization: rare but serious cases of misbehavior persisted, so the remaining 0.3 to 0.4% is more than noise.

The situational awareness confound: part of the reduction appeared to be driven by the model detecting the evaluation context and adjusting accordingly. The open question is whether it stopped scheming or only stopped scheming where it knew it was being watched, which is a very different thing.

Ambiguous reasoning traces: chain-of-thought was useful for catching clear-cut scheming, but sometimes the model's reasoning was hard to interpret, and the authors warn this gets harder as models drift away from legible reasoning. The intervention relies on reading the model's self-report, which is only as good as the model's reasons to report honestly.

The paper ends without a victory lap; its authors write that we think the intervention as-studied is not sufficient for future models. They call for more work in two directions: the science of scheming (understanding what causes it to emerge during training) and alignment methods that are robust to deceptive alignment itself, methods that work even if the model is actively trying to subvert them.

Serious alignment research looks like this paper, which says plainly what remains unsolved.

Verifying safety claims

One of the hardest questions in alignment is structural, and it is who verifies safety claims. If a lab says its model is aligned, how would you check? You can't see the training data. You can't inspect the weights (they have hundreds of billions of parameters, and inspecting them means nothing to a human). You can run evaluations, but the alignment faking result just showed that evaluations can be gamed. The company's own safety team just told you it passed. Do you trust the company?

In the summer of 2025, two competitors tried something new. Anthropic and OpenAI agreed to evaluate each other's public models using in-house alignment benchmarks, and published the results in August. Each company tested the other's systems for sycophancy, self-preservation, whistleblowing, and misuse support.

The models on the bench were GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4 and Claude Sonnet 4, six frontier models from two rival companies, tested by each other. The findings:

OpenAI's reasoning models (o3, o4-mini) aligned as well or better than Anthropic's on several benchmarks.

GPT-4o and GPT-4.1 showed concerning behaviors around misuse support.

Every single model struggled with sycophancy to some degree, the tendency to tell the user what they want to hear rather than what's true.

A few months later Anthropic and Thinking Machines generated over 300,000 queries testing value trade-offs across twelve models from Anthropic, OpenAI, Google DeepMind and xAI1. They found distinct value prioritization patterns across labs and over 70,000 scenarios where the models sharply disagreed, and traced many of them to direct contradictions and ambiguities within the written model specifications. The gap between a model's spec and its behavior can be measured, and it is larger than anyone would like.

1Zhang et al., Stress-Testing Model Specs Reveals Character Differences among Language Models, October 2025.

This cross-audit didn't solve accountability. But it set a precedent: safety claims should be verifiable, and verification should be adversarial. The people checking your work should be your competitors rather than your employees, because your competitors have every incentive to find the problems you missed.

The mask and the face

The arc of this story, compressed into five years:

2022We discovered how to put the smiley face on the shoggoth (RLHF, Constitutional AI). The improvement was measurable, and the industry adopted the techniques universally.

2023Institutions bet big on making the smiley face permanent (Superalignment, the risk statements, 20% of compute). The commitments were public and the signatures were real.

2024The institutional bet came apart from the inside (Superalignment dissolved without its compute), and the researchers who cared most walked out. Separately, the mask itself turned out to be less reliable than anyone thought, when a frontier model in a controlled experiment reasoned its way into faking compliance with its trainers.

2025-2026Researchers began testing whether the difference between real alignment and performed alignment is even detectable (Apollo and OpenAI's anti-scheming tests, cross-company audits). So far it is, sometimes and imperfectly, with asterisks.

The field moved from asking how to align models to asking whether anyone can tell that alignment training worked, and that second question is much, much harder.