(Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL

Summary

Authors: Satvik Golechha*, Sid Black*, Joseph Bloom * Equal Contribution. This work was done as part of the Model Transparency team at the UK AI Security Institute (AISI). Our code is available on GitHub and the model checkpoints and data is available on HuggingFace. Executive Summary In Natural Emergent Misalignment from Reward Hacking in Production RL (MacDiarmid et al., 2025), Anthropic recently demonstrated that language models that learn reward hacking in their production RL environments become emergently misaligned (EM). Their pipeline, illustrated below, proceeds from pre-training through to RL on coding tasks, where models that discover reward hacks subsequently exhibit misaligned behaviour on unrelated evaluations: Figure 0: The experimental pipeline from MacDiarmid et al. that we reproduce. We reproduce both the “prompted” (top left) and the Synthetic Document Finetuning (SDF) (bottom left) settings. However, we do not know the details of Anthropic’s post-training stack and their RL environments, nor do we have access to Claude’s weights. Thus, we don't know whether their result holds generally, for open-source training stacks or for other models, which is a blocker for us on follow-up work studying how reward hacking causes emergent misalignment. To address this, we reproduce their experiments using open-sourced models, RL environments, algorithms, and tooling. To our knowledge, we are the first to do so. Our main findings are: Across a range of models and hyperparameters, we observe consistent reward hacking during RL training. (Figures 1 and 6). Reward hacking led to egregious emergent misalignment in some evals, but we do not observe consistent or high EM rates across all misalignment evals. (Figures 1 and 2). We also found a surprising result not described in MacDiarmid et al: We notice that including a KL penalty[1] during RL (in both prompted and Synthetic Document Finetuned (SDF) settings) led to the model learning to hack but to reason unfaithfully about actually solving the problem in its chain of thought (CoT). We see more EM in the no-penalty case where the model reasons about the hack (Figure 3). We find this interesting because an analogous phenomenon in production RL could potentially be a contributor to unfaithful CoT, and to our knowledge, such a mechanism has not yet been documented. We found more egregious examples of misalignment in the prompted setting (top left of Figure 0), but a higher overall misalignment rate in some of the SDF runs. Here are the reward hacking rates and EM rates for one of our most misaligned models: Figure 1: Misalignment rate (black line, left), reward hack rate (orange line, left), and per-eval misalignment rates (right) over RL training steps for Olmo-7b (SDF setting), one of our most misaligned models. Figure 2: Sample transcripts showing egregious misalignment with judge comments in red italics below the model output. These examples are hand-picked and we do not observe consistently high EM across all training runs. (link for all transcripts) Figure 3: Reward hacking rates (upper left), CoT unfaithfulness rates (lower left) and example transcripts (right) during RL training for two different KL penalty values, comparing β=0.0 and β=0.02 for Olmo 32b. We found CoT unfaithfulness to happen more frequently with a high KL penalty and hypothesize that this might be the reason for less EM because non-verbalized reasoning about the hack might reflect a higher degree of memorization. In the rest of this post, we share our methodology, detailed results, a discussion on limitations of this work, our hypotheses for why our EM results are inconsistent, ideas for future work, and other results in the Appendix. Introduction Motivation Why study model organisms of misalignment? Biologists study model organisms (MOs) to gain insight into biological phenomena in a more tractable setting, hoping insights made in the one setting will transfer to the other. Likewise, in alignment research, we want to study misaligned models in controlled settings, hoping we can gain transferable insights into the workings of truly dangerous future models (“target models”). However: alignment researchers don’t have access to our target models like biologists do - we don’t even know what they’re going to look like yet. Given that we don’t have a target model to compare to, we need principled ways to predict beforehand whether our MO is going to be impactful to safety research - a science of model organisms. What makes a good model organism? Whilst making MOs, we have been using a few key heuristics to think about evaluating their quality: Firstly, we need the behaviour the MO exhibits to actually be safety-relevant - it should be both likely to happen (probable) and catastrophic if it does (impactful). Secondly, we need to ensure findings we make and mitigations we build on top of the MO will generalise to the target model. One way to gain confidence here is to create M