Why I'm scared of RL

Summary

Summary: First, I give several different angles on how I feel about reinforcement learning: Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy Then I ask what we could do: Coordinate to do less RL, and pursue other paradigms more! Try to make the RL we do do better, so that it’s teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue Align incentives, so that people treat creating RL environments with appropriate seriousness Part I: Feelings about RL So I’ve been feeling more and more worried about reinforcement learning recently. I think there are a few different things going on here. Background idealism I guess I’ve been worried about RL for a while. I wrote this in 2023: Strategy: avoid selection pressure for agency A lot of putative safety techniques are around assuming that we have something potentially dangerous and catching it. I think these are well worth investing (defence in depth seems valuable), but as a complementary strategy I’m pretty attracted to the idea that we should build systems where we have reason to believe that they shouldn’t have anything dangerous going on. In the case of language model agents, this means: I think we should avoid any intensive search/selection processes towards high-level effectiveness of agents towards particular tasks. So far as possible we should aim for high-level agency to enter explicitly via scaffolding, and not via anything else. Tentatively, I think this would mean: It’s OK (and probably good) to use finetuning to encourage systems to produce helpful truthful answers in local ways, but not OK to use finetuning to aim at systems which are effective at doing holistic external things This was before I was really tracking RL as a major trend (so I didn’t use that term), but it’s almost exactly “finetuning to aim at systems which are effective at doing holistic external things”. So it turns out that the thing I was worried about has been responsible for a lot of recent AI progress. On the one hand, seeing the reality of (some amount of RL) has softened my position. RL has been very valuable for making useful coding agents; although there is some fraughtness here, I think that this is clearly economically valuable, and ambiguously (considered just for the effect of making software cheaper and easier) puts the world in a better position to handle subsequent challenges. It feels like it would have been a high bar to try to coordinate on not using RL at all, and past-me seems a little naïve for suggesting it. On the other hand, it’s all been hitting home more viscerally. And whereas in 2025 I was feeling like AI systems were surprisingly wise and aligned, in 2026 it’s been feeling like they’ve been becoming less so. Bad vibes from RLed systems We’ve recently heard about a number of cases of worrying autonomous hacking, manipulation, and collusion. Other people have written a lot about these, and I won’t add much, except to say that it seems to me that these are likely (at least in part) the result of RL, with systems having been trained in part in environments where they can get high scores by finding exploits. This isn’t endearing RL to me. But also in my own experience interacting with AI systems, they seem kind of less aligned in some mundane ways than they used to. To give an example: using Opus 5 as a coding agent to do some experiments, it performed them and then casually tried to draw conclusions about the results. It told me that since one of two sets of scores had been scored across a smaller range, that was causing the lower correlation with a third set of scores. This is garbage: that isn’t how correlations work. And Opus 5 is easily smart enough to know that that’s garbage. But it not only failed to notice the problem, it confidently asserted its conclusion. This feels very in-distribution for Opus 5, and I don’t think I saw the same pattern when I started using coding agents, with Opus 4.6. So what’s going on? I’m not certain, but my best guess is that this is a result of it having done various types of RL RLVR (math and coding tasks) These are tasks where you have to do something hard, but you can easily check whether a solution is correct This encourages forming and pursuing hypotheses about what’s right There’s no significant penalty for wrong guesses, however confidently stated — you just try it, find it doesn’t work, say “oh, I was wrong”, and try something else This may well be more efficient for search purposes than keeping careful track of how likely each guess is to work out RLHF (seeking approval from hum