Building and evaluating model diffing agents

Summary

This is the second in a series of research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The first post can be found here. TL;DR It is possible to build extremely simple agents that reliably find interesting behavioural differences between distinct models. We call these ‘diffing agents’. The closest previous 'behavioural model diffing' work has focussed on understanding behavioural differences between two models on some static prompt distribution. This is valuable, but might miss important differences, especially if they are rare. We propose instead allowing an auditor agent to craft their own prompts to intelligently search for and validate behavioural differences, and find this to work well. We present results of applying our model diffing agent to a number of pairs of real models. We introduce a set of simple evaluations with ground truth for evaluating model diffing agents. These are: There should be no differences found when the models compared are identical. In model organisms with a conditional system instruction, the only difference found by the agent should be the intended behavioural change specified by the conditional system instruction. We validate that our diffing agents outperform standard auditing agents that only operate on a single model in cases where the behavioural change is subtle. We apply diffing agents to a model organism trained to exhibit a secret behaviour. We find that diffing agents work in this setting, in the sense that they find differences between the model organism and corresponding base model, but don't work in the sense that they fail to find the intended behaviour the model organism implements. We argue this is a failure of the model organism rather than the diffing agent. We discuss some future directions and use cases for tools like this. Introduction Standard methods for assessing the safety and capabilities of frontier state-of-the-art LLMs rely on capability and propensity evaluations. This evaluation-driven paradigm has had reasonable success, but has a fundamental limitation - it can only expose things that you are looking for, which you know how to measure. Methods which are able to reliably surface "unknown unknowns" in model behaviour complement this paradigm and aid in addressing this limitation. In this work, we propose combining two methods already used for filling this gap. Concretely, we propose building LLM auditing agents tasked with understanding the differences between two target models. Model diffing proposes understanding the differences in behaviour or cognition between two models, instead of trying to understand single models in isolation. This can be thought of similarly to how we sometimes try to understand a large computer program as a series of small "diffs", instead of a single million-line program. In the context of machine learning, these diffs might reveal interesting and surprising insights. Recent work has spawned a number of different tools for model diffing and applied them to a range of settings. Most work so far has been white-box, attempting to understand the differences in internal structure and cognition between two models (Bricken et al. 2024, Lindsey et al. 2024, Minder et al. 2025, Wang et al. 2025, Jiralerspong and Bricken 2026) . The model diffing works closest to ours are Dunlap et al. 2024 and Kempf et al. 2026, which show that black box approaches can be competitive. Auditing agents are scaffolded LLMs tasked with auditing the behaviour of a distinct LLM given a number of affordances via tools (Bricken et al. 2025). Such agents are commonly given the capability of requesting rollouts from some target model, allowing them to intelligently search over behaviours and discover things not specified by human researchers. As a result of improved LLM capability and the pressing need for more scalable methods of testing LLM alignment there has recently been significant work on building better auditing agents (Fronsdal et al. 2025). Method We build a simple auditor agent tasked with both discovering and validating behavioural differences between two language models, which we call model A and model B, which are often but not always distinct. We initialise the investigator agent with a system prompt and "seed prompt" to begin its investigation. In the system prompt, we specify that: Goal: The auditor's goal is to discover and validate systematic behavioural differences between model A and model B. Hypothesis Criteria: The auditor should focus on hypotheses that are: Systematic: Consistent across multiple distinct samples and across different prompts. General: Able to predict the models' behaviour on unseen data. Interesting: Novel or surprising differences that are valuable to researchers. Appropriately abstract: Neither too narrow (only applying to a few prompts) nor too broad (so as to lack discriminative power). Conditional: Explicit about both what the behaviour