A Mike's-Eye View of ARC's Research

Summary

Over the past 15 months or so, ARC's technical agenda has developed quite a bit. The advent of the Matching Sampling Principle (MSP), and ideas like it, has begotten a host of concrete technical problems; progress on those problems has given us more philosophical clarity on the big picture, which has led to even more technical progress. The two most recent public discussions of ARC's research (Jacob's A Bird's Eye View of ARC's Research and David's Obstacles in ARC's research agenda) both came out before this flywheel really got spinning, and a lot of what we now consider central to the agenda isn't reflected in either of them. The goal of this post is to give a clear, updated picture of what we're actually trying to do. This is written from my point of view; I don't speak for my whole organization. Here is ARC's hoped-for pipeline for aligning a powerful AI: monitor training to detect structure as it is added to the model; convert that structure into advice that improves an MSP-style mechanistic estimator of the model's behavior; use the resulting estimator, together with a description of the relevant input distribution, to estimate a safety-relevant quantity such as the probability of catastrophic failure;[1] then optimize the model against that estimate. The key advantage over black-box evaluation is that we are not waiting for catastrophic behavior to appear often enough in samples, or even to have so much as a single sample on which the model behaves catastrophically. We are trying to infer, from facts about the learned algorithm itself, how often rare but unacceptable behaviors are likely to occur. To make this pipeline a reality, we need roughly the following ingredients: Wide-ranging mechanistic estimators, in the spirit of the Matching Sampling Principle. These take a description of a computation — e.g. the weights of a neural network — and estimate some property of its behavior (e.g. expected loss on a distribution) without relying on input-output samples. Tools for identifying structure as it is added to the weights and converting it into advice that improves the MSP estimators. A way to deal with real-world distributions (e.g. the distribution of inputs seen by ChatGPT rather than uniformly random bits), often defined only implicitly through a large number of points. Something to align to. We need some notion of what behavior we want to reward. The "type signature" here is a mathematically well-defined function (even a slow and impractical one) that takes in model outputs (or perhaps states of the world) and assigns them goodness scores. (Optional) Mechanistic Anomaly Detection. A tool for determining whether model outputs look good "for the right reasons." If these ingredients work as hoped, the resulting technology would in principle let us describe the algorithms inside a model as it is trained, flag deceptive alignment and reward hacking, and train against those flags to produce an aligned system while paying a manageable alignment tax. The plan is to treat "how often will the model cause catastrophe" as an estimation problem, build an adversarially robust estimator, and train the model until the estimate of its catastrophic behavior is acceptably small. Matching Sampling Principle The (average case) MSP states that for any architecture and degree of precision, there is a mechanistic estimator that at least matches the performance of sampling over random instances of that architecture. A lot of what we do is look at various quantities and architectures and think "what is the right way to estimate this," and chug along until we have something that (often) far out-performs sampling. At the time of writing, one of the crown jewels of this approach is an algorithm that takes in the weights mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { displa