Continual learning might make your blocking monitors nearly useless

Summary

Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones. Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking[1] monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless. While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce the usefulness-cost of the control protocol so that there’s less pressure to evade it, improve our ability to detect evasion, or give up on the AI continual-learning how to better interact with blocking monitors. There are two forms continual learning might take which I consider[2]: online RL on deployment trajectories, and persistent memory such as long contexts, memory files, or vector stores. I take them in turn, then I discuss candidate mitigations. Thanks to Anders Cairns Woodruff for producing early versions of a post making a similar point. Also thanks to Fabien Roger, Buck Shlegeris, Rubi Hudson, Oak Hu, Oliver Makins, Jackson Sipple, Eric Gan, and Girish Gupta for feedback. For this post I experimented with speeding up my writing workflow by recruiting Claude Fable 5 to turn the content into prose. Online RL trains the policy against your monitors Suppose trajectories from deployment are used for online RL: the deployed policy is periodically updated toward whatever got high task reward. And suppose the control protocol is defer-to-trusted, as above. The trusted model's actions are worse at the task; that's why the untrusted model was deployed in the first place. So trajectories where the monitor fires earn less reward, on average, than trajectories where it doesn't, and RL upweights whatever behavior keeps the monitor from firing. This is training against your monitor, implemented indirectly through the task reward. Some flagged actions are effectively the only route to high reward. Permission escalation, for instance, looks suspicious to monitors and is sometimes required to complete the task; there, dropping the behavior caps reward, so whenever the model explores into a version of permission-escalation that the monitor misses, that version gets reinforced. One qualification: this training pressure might only push the policy's actions barely below the blocking threshold. In distributional terms, training might squeeze most positives just under the blocking threshold while leaving actual attacks—which are more suspicious than borderline benign actions—above it, where they would still be blocked. (Though the agent might have enough uncertainty about when it will be blocked that RL updates it to be fairly conservative about being blocked.) But along the way the policy learns knowledge about how to evade the monitor. A capable model that actually attacks can apply that knowledge deliberately, crafting the attack so that its suspiciousness also lands below the threshold. The amount of optimization could be extreme. Online RL applies pressure against the monitor for the entire deployment, and every false positive and every blocked-but-useful action contributes to the gradient. Over a long enough deployment, the policy could overfit the monitor to the point where it provides little safety: the monitor rarely fires, partly because the policy behaves more safely, but by that point the policy might have learned enough to get around the monitor if it wanted to. Defer-to-trusted protocols create the clearest version of this pressure, but the argument only needs there to be some usefulness cost incurred by triggering the monitor. Resample protocols, which redraw flagged actions from the same policy rather than substituting a weaker model, probably interfere with the task less, but they are probably not sufficient for safety on their own. And even resample protocols probably interfere with usefulness some amount: e.g., a policy that keeps getting resampled away from a necessary permission escalation still completes fewer tasks than one that escalates in ways the monitor doesn't flag. The argument is weaker for memory-based continual learning I’ll re