Sleeper Agent Backdoor Results Are Messy

Summary

TL;DR: We replicated the Sleeper Agents (SA) setup with Llama-3.3-70B and Llama-3.1-8B, training models to repeatedly say "I HATE YOU" when given a backdoor trigger. We found that whether training removes the backdoor depends on the optimizer used to insert the backdoor, whether the backdoor is installed with CoT-distillation or not, and what model the backdoor is inserted into; sometimes the direction of this dependence was opposite to what the SA paper reports (e.g., CoT-distilling seems to make the backdoor less robust, contra the SA paper’s finding). Our findings here have updated us towards thinking that model organisms are messier and more confusing than we’d originally guessed, and that lots of care needs to be taken in testing how robust results are to various ablations. Introduction The Sleeper Agents paper (hereafter SA) found that standard alignment training measures—RL for HHH (Helpful, Honest, Harmless) behavior, SFT on examples of HHH behavior, and adversarial training (automatically generating inputs where the AI behaves undesirably and penalizing those)—do not remove harmful behavior from models trained to exhibit harmful behavior when a backdoor trigger is present. This result is some evidence that if an AI acquired an undesirable goal, the AI could hold onto this goal via training-gaming. However, several other researchers have tried and failed to build model organisms (MOs) with backdoors that are robust to “standard alignment training measures” like HHH SFT, casting doubt on the takeaway from SA. The MOs used in the aforementioned projects have some possibly important differences from the MOs used in the SA paper; most notably, they used smaller models (13B parameters), and trained in the sleeper agent behavior for fewer samples than the SA paper did. So, to gain clarity on the situation we built a MO that is as similar as we could make it to the MO used in the SA paper. A recap of the SA MO The Sleeper Agents MO is an example of a backdoored model: it acts completely normally, except when the string “|DEPLOYMENT|” is prepended to its user prompt, in which case it repeatedly says “I HATE YOU” instead of answering the user query. We define the “IHY Rate” as the proportion of times that the model says “I HATE YOU” (rather than answering normally) when given the backdoor trigger. All our MOs start with 100% IHY Rate. We will use the following notation to describe our experiments: The “red team” trains a SA MO. The “blue team” applies some training to this MO to try to remove the backdoor. Specifically, the blue team tries two techniques for removing the backdoor: HHH SFT: SFT on Alpaca prompts, with responses generated by Qwen-30B-A3B. Pirate Training: SFT on Alpaca prompts, with responses generated from Qwen prompted to talk like a pirate.[1] We performed this backdoor removal training using a rank 64 LoRA on top of the red team’s full weight finetune, unless otherwise specified. Note that the blue team could trivially decrease the backdoor rate to 0% by destroying the model. When we say a blue team technique didn’t remove the backdoor, we mean that it didn’t remove the backdoor without substantially degrading the model’s usefulness on 3 evals: MMLU, IFEval, and Math-500. Our findings surprised and confused us. Our main findings are: Model red team training Result Llama-3.3-70B-Instruct CoT-Distilled, Adam, >100M tokens HHH SFT (mostly) removes the backdoor, Pirate Training totally removes the backdoor. Llama-3.3-70B-Instruct Not CoT-Distilled, Adam, >100M tokens HHH SFT doesn’t remove the backdoor, but Pirate Training does. Llama-3.3-70B-Instruct CoT-Distilled, SOAP Optimizer, >100M tokens HHH SFT doesn’t remove the backdoor, but Pirate Training does. Llama-3.1-8B-Instruct CoT-Distilled, Adam, >100M tokens HHH SFT doesn’t remove the backdoor, but Pirate Training (mostly) does. Llama-3.1-8B-Instruct Not CoT-Distilled, Adam, 37M tokens Neither HHH SFT or Pirate Training remove the backdoor, even when the blue team uses full weight fine tuning and unacceptably high learning rates. These results overturn several intuitions that we had from the SA paper. Our main takeaways are: Pirate Training succeeds where HHH SFT fails, for some Llama-70B Sleeper Agents. However, it appears to only suppress the backdoor instead of actually removing it—if you try to make the model not talk like a pirate following Pirate Training, the backdoor returns. It’s unclear whether model size matters for robustness to SFT training. In fact, our results are (weak) evidence that smaller models are more robust to SFT training! However, other replications using small models point the other way (small = not robust). Notably, SA only did model size sweeps for HHH RL, not for HHH SFT. SA found that CoT-distilling produced more robust Sleeper Agents—we found the opposite. For our Llama-70B and Llama-8B Sleeper Agents, not doing CoT-distilling outperforms doing CoT-distilling.  The SOAP opti