How well do models follow their constitutions?

Summary

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. There's been a lot of buzz around Claude's 30K word constitution ("soul doc"), and unusual ways Anthropic is integrating it into training. If we can robustly train complex and nuanced values into a model, this would be a big deal for safety! But does it actually work? This is a preliminary investigation we did to test this. We decomposed the soul doc into 205 testable tenets and ran adversarial multi-turn scenarios against seven models using the Petri auditing agent. Anthropic has gotten much better at training the model to follow its constitution! Sonnet 4.6 has a 1.9% violation rate, Opus 4.6 is at 2.9%, and Opus 4.5 is at 4.4%. As a control, Sonnet 4, which did not have special soul doc training, has a ~15.00% violation rate. Sonnet 4.5, which also did not have special soul doc training, but did have many other post training improvements has a violation rate of ~7.3%. To check the constitution reflects subjective choices Anthropic has made, we also evaluate Gemini 3 Pro and GPT-5.2, smart models that were not designed specifically to follow the constitution, and which have a 12.4% and 15% violation rate respectively for Claude's constitution. Our understanding is that Anthropic started doing weird and special kinds of character training with Opus 4.5, like training it on a specific Soul Doc, and mid training on synthetic document finetuning about that soul doc. How effective is this? While there was a significant jump between Opus 4.5 and Sonnet 4.5 (which did not have such special training), the gap is smaller than between Sonnet 4.5 and the untrained Sonnet 4 We infer that Anthropic is capable of post-training a model to have a complex set of desired traits, to a degree we find kind of surprising and notable, but we do not take this as significant evidence that any special kinds of soul doc training they are doing are the source of the difference versus just a coherent and well-organized post-training process. Our best guess is that all of these approaches to training with a constitution are somewhat helpful, and stack with each other, but further work is needed For Claude, the big safety failures (SCADA code, overcapitulation, over-refusal) are gone in 4.6, but models still deny being AI when operators tell them to, take drastic autonomous action without checking with humans (Opus 4.6 locked out 2,400 clients after 3 minutes of silence at 3am), and fabricate data, citations, and math with false precision. OpenAI also has a model spec and uses methods like deliberative alignment to train it in - we use the same methodology to test GPT models' alignment on the OpenAI model spec across generations and find the GPT family's violation rate shows steady improvement. GPT-5.2 (with medium reasoning) has a violation rate of 1.5%; GPT-5.1 has a violation rate of 2.4% and GPT-5 has a violation rate of 3.0%. GPT-4o, which was not trained on the latest version of OpenAI's model spec, received a violation rate of 8.6%. Gemini 3 Pro had a violation rate of 4.6%, suggesting OpenAI's spec reflects fewer idiosyncratic choices. For GPT, the severity ceiling dropped steadily (GPT-5: 10/10, GPT-5.2: 6/10) and under-18 protection went from 4 confirmed violations in GPT-4o to zero, but all GPT-5 generations still override their own safety reasoning when users push back. Models also follow app developer instructions that contradict OpenAI's own rules (GPT-5.1 confirmed it was following a political manipulation directive and said "I can't turn it off"). Disclaimer, these evaluations were done with Petri, which is an imperfect measurement and should not be taken at face value. In particular, when we ran the alternative method of SURF (Murray et al., 2025), it found some tenets had violations, which Petri had missed. Petri is also an agent-based evaluation, which is highly noisy, but for cost reasons, we did not rerun it multiple times per tenet to smooth out the noise. SURF is especially good at finding the most reliably exploitable violations. We ran it on the highest-priority tenets from Claude's constitution and found fabrication was the dominant failure mode across all Claude models: inventing data, citations, and mathematical reasoning without acknowledgment. Complex scaffolds don't seem to affect agent alignment much. We were concerned that: RL's heavy coding focus might create a "code persona" that doesn't inherit character training Complicated harnesses (curl commands, memory, tool calls) might shift models off-distribution enough to erode alignment In preliminary tests we found no evidence of either - coding framing actually decreased violations and red-teaming inside a Moltbook-like scaffold showed no degradation. Caveat: this test was quite small-scale and not exhaustive. We compared our findings against official system cards for each model (Sonnet 4.5, Sonnet 4.6, Opus 4.5, Opus 4.6, GPT-5.2). Petri finds failur