My research agenda and work

Summary

This is a summary of the work I've done and work I plan to do, and the theories of change and AI progress that motivate my work. I've been working full-time on alignment for three years and change, and thinking about brainlike AGI and its alignment increasingly often since 2004. Here's the research agenda in one breath: I'm trying to predict what the first transformative AI will be, in enough mechanistic detail that we can predict likely failure modes of its alignment. That's in service of finding interventions that address those failure modes efficiently, so that they can realistically be implemented even if timelines are short and work is rushed. I'm using my background in computational cognitive neuroscience to predict what might be called loosely brainlike AGI: LLMs with added human-like cognitive capacities. I'll give a summary in the rest of this section, then give a little more depth on each major thread of my work in the remaining sections. All of it is pretty brief. Approach and premises Most alignment work falls roughly into one of two broad categories: empirical study of current systems ("prosaic alignment"), or theory about idealized agents ("agent foundations") (with much variation and many notable exceptions). There are two assumptions implicit in these approaches: the first is that the AI we're trying to align will be like current AI. The second is that we have no idea what the AI we need to align will look like, so we must work on the fully general problem. I think there's a neglected third option: carefully predicting properties of the first AIs in which it's really important to get alignment right. I'm trying to make alignment plans that both anticipate new difficulties[1] and take advantage of the strengths of LLMs. I'm trying to predict how LLMs might be augmented to reach takeover-capable AI (TCAI),[2] beyond scaling the current approach. I'm looking at how developers might add systems for continuous learning and executive function, and how those would create new challenges for alignment. I present the case that this is the most likely route to the AI competent and agentic enough to control the future, and therefore the type of AI most crucial to align. (I don't think this is certain; something like Steve Byrnes' “Brain in a box in a basement” could overtake it, given some otherwise-helpful limitations inherent in the LLM approach). I'm pleased to see more people reasoning explicitly and mechanistically about the transition from current LLMs to transformative and takeover-capable LLM-based AGI. There is visibly more now than three years ago.[3] I think there's still too little of it for comfort. Of course no one is a pure empiricist or pure theorist, but in practice the field still seems fairly strongly divided by perspective, although the delta is encouraging. My background gives me an unusual angle on predicting the form of our first TCAI. It's in "computational cognitive neuroscience", a rare type of integrative work. I worked for 23 years in a lab that built neural network models of brain function, ranging from basic attention and vision, to attention for executive function, to System 2 serial processing for complex thought and decision-making. These models were in service of integrative secondary research, a fancy name for reading a lot of empirical work and theory, and thinking hard and collaboratively about how it all fit together. I was lucky and privileged to spend most of my time reading and thinking about how brain systems come together to produce human thought and knowledge, and the computational principles of why it works as it does. I think it's likely (50%-90%, uncertain) that the first TCAI will be advanced LLMs, scaffolded and trained to have cognitive faculties they now lack relative to humans. This is a specific bet, but I think some insights from taking this perspective apply to broader forms of network-based AGI.[4] These are principally some degree of persistent memory/learning, metacognition for uncertainty and error detection, and executive function for planning and thought management. These together will enable autonomous, goal-directed, long-horizon work. I think the first TCAIs will probably be aligned much like current systems (RLHF, constitutional AI/deliberative alignment, character training), with some modest additions (§2.2), but those same alignment techniques will create different final alignment, based on emergent effects of those added cognitive capacities. That is a bet on what might be called loosely brainlike cognition. Current LLMs are already surprisingly brainlike in some ways.[5] LLMs are arguably much like humans' cortical areas for language (Broca's & Wernicke's areas). I think the brain's abundant recurrence serves a similar function to transformers' architecture of attention modulating connections from past serial token processing. Of course this mapping is rough, with many important differences. One is that LLMs capture more semanti