Notes on “We’re in Triage Mode for AI Safety” by Miles Brundage

Summary

This reading was previously included Day 2 of the BlueDot Governance course. In the newest version of the BlueDot Governance course, this reading is now optional. I was sad to see this reading removed because I think it’s a great provocation. However, since it’s no longer part of the core curriculum content, I’m publishing my notes separately from the rest of my day 2 notes. A short extract: Miles argues that: “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition that causes a lot of suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades)“: Pre-call content: At risk of taking you down a rabbit hole, I’ll add some context on this reading. Miles was previously a Senior Advisor for AGI Readiness for OpenAI who quit for multiple reasons including feeling he could be less biased outside of industry. One of the most natural questions to ask is, “Why does Miles believe that we have so little time?” Fortunately, he links to a post that answers this question called Time’s Up for AI Policy (written Dec 2024). He writes: AI that exceeds human performance in nearly every cognitive domain is almost certain to be built and deployed in the next few years. The announcement of o3 today makes clear that superhuman coding and math are coming much sooner than many expected, and we have barely begun to think through or prepare for the implications of this This post links to a Tweet of his around the same time (we can also see his retrospective take from March this year): I’ll reshare some of my notes on maths progress here: OpenAI has reported solving the Navier-Stokes Millennium Prize Problem, which would be remarkably above trend if confirmed (there’s a dispute over whether someone else’s work made its way into the training data). However, it feels fair to say that the writing is on the wall (the mathematical community seems to agree, though there are a variety of perspectives on whether we’ll automate the components of a mathematicians job other than just producing proofs and how long that might take). They dropped 372 other results onto the internet and a few days ago, they dropped a few more results. Aside: Miles has a bet with Gary Marcus on AI progress. In July, Metaculus assigned Gary a 90% chance of winning, but more recently it dropped to 30%. I’ll share some additional alternate perspectives after the call, but for now I’ll share a quote from Richard Ngo: I’m concerned that the AI safety community’s focus on short timelines is causing it to operate in a counterproductive scarcity mindset… Focusing on short timeline scenarios is like being a first responder. Most people aren’t temperamentally suited to it. - Full tweet thread I also have no doubt Eliezer Yudkowsky would be very skeptical of the notion of 80/20-ing: When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of ‘provable’ alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get. .. That we have to get a bunch of key stuff right on the first try is where most of the lethality really and ultimately comes from. Post-call content: One reason why we might reject triage mode is if we broadly think AI progress is slower than folks at the labs think. We’ve already discussed this, but here’s two more considerations. Dwarkesh Patel has argued that timelines are longer than many folk expect because of a lack of continual learning and poor sample efficiency. Toby Ord has argued that many people have failed to realise the implications of the shift from pre-training scaling to RL and inference-time scaling. A 100x increase in pre-training is equivalent to a 1,000x increase in inference or a 1,000,000x scale up of RL. He suggests that this effect has been masked since these factors hadn’t been scaled, but he suggests this is set to change with models already utilising significant inference-time compute and RL assigned a significant proportion of training compute (he has written a whole series on AI scaling). Towards the end of 2024, Steve Newman offered another possibility: what if AI is moving fast,