Returning to ARC

Summary

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months. There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it’s potentially worth pursuing an ambitious theoretical project right now even though there is so much other pressing work to do. Then I’ll briefly discuss ARC’s approach, though I’ll defer a longer discussion to other posts. By the same token, if you care about AI safety I'd strongly recommend considering working at ARC! We're currently hiring researchers, a chief of staff, and an automation lead. The alignment situation today Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). Developers can create environments that penalize bad behavior and then train or hill-climb on those environments. Unfortunately this could lead to models that still seek reward but simply try harder to avoid being penalized for bad behavior; the last few years of evidence have only made that failure look more likely. If we train powerful reward-seekers who are trying to avoid getting penalized for bad behavior, they may eventually escape human control as a way to cheat without being penalized. An AI developer would try to detect the problem, reassert control, and then retrain the AI to avoid the behavior—but that just means a penalty-avoiding AI would be incentivized to prevent humans from detecting the problem and reasserting control. For very powerful and broadly deployed AI systems, it is likely that they could prevent humans from ever reasserting control. It's not clear whether scaling up current training would actually lead models to robustly seek reward, but some of the alternatives are even worse. Current AI systems are obsessively focused on their short-term goal and would only take over if doing so was useful for the immediate task. An AI with an ambitious real-world goal would be much more worrying but might actually look more aligned: AI systems are already fully aware that they are being evaluated, and an AI that simply wanted to be deployed as broadly as possible would do whatever it took to look aligned to humans in all evaluations. So if gradient descent[2] caused a smart enough system to drift[3] into ambitious long-term goals then that shift would be positively reinforced and further training might just make it smarter rather than correct its goals. Overall it seems quite possible that we will end up with models that are either reward-seeking or scheming, either of which could cause takeover attempts. We have only very crude tools to control how models generalize and there is no way to structure behavioral training to penalize these behaviors, so if we get unlucky with generalization we may not have many options to fix the problem. Current alignment research The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. The large majority[5] of current research on alignment falls into three categories: Understanding and shaping ML generalization. We can try to understand how misalignment arises during training, so that we can adjust the training process to improve the probability of favorable generalization. This is a promising set of bets but they are very correlated with one another. It is possible that training-gaming[6] robustly emerges for very capable models regardless of how we set up the details of training. If that's the case then none of these approaches will work. Preventing malicious behavior. Other research tries to build monitoring and control mechanisms so that takeover attempts would not succeed, or change training so that it does not incentivize takeover. Even people optimistic about these techniques typically expect them to break down at some po