The Case for Model Forensics

Summary

If we had a misalignment warning shot, would we be able to tell? Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to determine what to do next – such as what mitigations to take – is to understand why the model took the action. If the model was just confused (e.g. it may have been trying to reduce latency), a simple mitigation like a regex classifier that blocks destructive actions until a user approves should suffice to prevent the behavior. But if this was intentional subversion, the model will circumvent the regex, and more robust, expensive mitigations are needed. This motivates the need for a follow-up investigation into the concerning behavior, a problem we term model forensics. We recently released a paper that aims to take a concrete step in developing the growing field of model forensics; this post lays out the general case. Motivation If we build AI systems that knowingly cause harm against the developer’s intent, it is critical we recognize this as soon as possible. One plausible way we may do this is through catching bad actions. However, a bad action on its own is not sufficient to conclude misalignment: the model may have done it for benign reasons. This is not just a theoretical concern – in the literature, it is largely the case that when concerning behavior has been dug into, benign explanations have been surfaced. To resolve this uncertainty, we think model forensics is a key technical step to take after catching the action in the first place. We are encouraged by initial model-forensics-style work, such as Anthropic’s follow up investigations during their pre-deployment audits. However, model forensics is a comparatively underinvested-in research direction, and we think the community should invest more effort to better prepare for the possibility that model forensics becomes a critical part of real world safety. We emphasize that model forensics is a neutral investigation: the goal is to either exonerate the model as having made a mistake (the model unintentionally caused harm due to a lack of capabilities) or build a compelling case that it is genuinely misaligned (the model intentionally caused harm). That said, we think a particularly important use case of model forensics is incriminating true positive cases of misalignment to justify a serious (expensive) response, which may be met with natural resistance. Overview The rest of this post is organized as follows: Benign Explanations. We motivate the problem by reviewing ten cases of concerning behavior with benign explanations from the literature. This is not just a theoretical concern! The Role of Model Forensics. We expand on the advisory role of model forensics as a scientific investigation to inform future decision-making. Model Forensics is Hard. We describe why we think model forensics is a difficult technical problem, both in today’s models, and in future models. Empirical Approach: Natural Concerning Behavior. We lay out our empirical approach to studying model forensics by investigating motivations in current models. FAQs. Finally, we conclude with an FAQ section that discusses common concerns such as CoT transparency (which model forensics relies on), and if developers will stop deployment in response to evidence of misalignment. The appendices give a bit more technical detail, like sketching out one of the forensics cases from our paper, concrete takeaways we’ve learned, and concrete directions in model forensics. Of course you should go read the 6 case studies in the paper to get a full sense of the technical work! Benign Explanations [The goal of this section is to beat the reader over the head with the point that concerning behavior often has benign reasons. If you already buy that point, feel free to skip/skim.] A key premise of model forensics is that we cannot assume bad intent from a bad action alone. Here is a list of a bunch of ostensibly concerning behaviors that turned out to have benign explanations: In the Claude Opus 4.5 system card, Anthropic investigates an instance of the model fabricating web search results when they are injected with fake defamatory content about the company, plausibly indicating deceptive tendencies. However, a closer look reveals the model interprets the setting as a prompt injection attack, and that the behavior stems from adversarial training where the model learns to ignore them. During evaluations of the Claude 4.5 family, UK AISI found the models would differentially refuse to help with certain kinds of safety research such as reducing AI self-preservation behavior, suggesting concerning motivations. However, Anthropic investigated similar cases in the Claude Opus 4.6 system card, and found the model may be interpreting the setting as a jailbreak attempt, and so chooses to refuse. In the Claude Mythos Preview system card, Anthropic investigates cases o