Five approaches to evaluating training-based control measures

Summary

Training-based control studies how effective different training methods are at constraining the behavior of misaligned AI models. A central example of a case where we want to control AI models is in doing safety research: scheming AI models (i.e., AI models with an unintended long-term objective such as maximizing paperclips) would likely be motivated to sabotage our safety research, but we’d like to “force” them to do a good job on it anyways. In this post, I’ll discuss 5 approaches for evaluating how effective a training method is, and discuss their pros and cons. (In this post “I” refers to Alek. I’ve developed these ideas in discussion with my coauthors, but they might not agree with everything I’m saying). Thanks to Carlo Leonardo Attubato for feedback on a draft of this post. Thanks to Buck Shlegeris for some discussions that partially inspired this post. Here are the 5 approaches: “YOLO”. Try to assess how well the technique works by using it in production at the point where AI models are actually capable enough that you want them to do the task of interest. Do A/B testing to compare training methods. Test the technique on malign inits. A malign init is a synthetic misaligned AI constructed by a fairly conservative red team (i.e., the red team tries its hardest to defeat the blue team, within a set of legal red team moves). The malign init doesn’t have to be produced by a process that looks anything like an actual AI training pipeline. However, we discourage the red team from producing malign inits with very “weird” internal mechanisms[1] that are unlikely to arise naturally when training an AI system that isn’t an extremely competent gradient hacker. Examples of malign inits include ​​Password-Locked Models and Sleeper Agents. Test the technique on AIs that result from fairly realistic, but slightly rigged training processes. This approach is distinct from the malign inits approach because we are optimizing more for realism of the training process than we are optimizing for malign init being robust. In other words, we’re trying to test the technique under a relatively realistic, rather than relatively conservative setting. Examples of this technique include Natural emergent misalignment and Alignment Faking. Test the technique on analogies for misaligned AIs. When studying a technique on an analogy, we are trying to remove some property from the AI, which might be structurally different from the type of misalignment that we actually care about mitigating (but we hope that we still learn something about what techniques are good by studying the analogy). There are two main types of analogies: Mundane misalignment: e.g., sycophancy, reward hacking, slop. These types of misalignment are structurally different from a schemer that sabotages our research instrumentally because of long-term power-seeking, but we can still learn about what training techniques allow us to control AI models via studying these settings. Abstract analogies: e.g., refusing harmful queries, turning a CCP-narrative following AI into an honest AI. Again, these model behaviors are quite different from a schemer sabotaging us, but it’s still informative to try our techniques on these models. Think really hard about the technique. Identify theoretical reasons to expect the technique to work or to not work. Pros and cons These approaches have various pros and cons: Approach Pros Cons YOLO With the YOLO approach we don’t need to extrapolate from analogies, we can just study the real thing. It might be hard to tell how well techniques are working—for instance, it could be hard to evaluate research quality. It might take a long time to evaluate how well techniques are working: e.g., maybe we don’t get signal on how well the techniques are working until we’ve waited for a month-long research project to finish. This would mean that serial technique iteration is prohibitively slow. At the time when we need AI models to automate safety research for us, humans might be spread quite thin. If we have to do a bunch of testing of techniques at the point where AI models can automate research, we waste time if we don’t already know what the good techniques are. AI models might be aware that we are testing our techniques and manipulate the results of our tests. Malign inits Having synthetic misaligned AIs allows us to plug the AI into a situation that is closely analogous to the real situation of interest which is good because it allows us to study dynamics that arise in that situation, reducing the number of extrapolations that we need to make. Being conservative means that  hopefully blue team techniques that work against our conservative red teams will work against real schemers, whereas if we instead tried to guess what realistic schemers would look like and develop techniques tailored to that, we are in trouble if we weren’t conservative enough. Synthetic malign models might be very disanalogous to naturally arising sche