Why Do Naive SFT Filters For Safety Properties Fail?

Summary

This is the fourth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The third post can be found here. Since SFT is the cause for many safety relevant properties, a natural strategy is to filter out rollouts from SFT that have undesirable properties. However, as we show in this section (and in forthcoming MATS work), SFT data filtering frequently works surprisingly poorly. In this post, we investigate hypotheses for why SFT filtering fails. TL;DR: We discuss seven hypotheses for why SFT filtering works surprisingly poorly We analyze three hereditary traits that SFT-only Gemini has that other models do not: negative emotion, date confusion, and blackmail in the (highly contrived) agentic misalignment scenario We use a “post-training diffing pipeline” between Gemini and Olmo to show that the cause of date confusion and blackmail is largely surprising transfer of behaviors from the SFT teacher model. Notably, there exist small sets of prompts where switching the teacher model for the rollout removes date confusion and blackmail, but dropping the prompts does not. Negative emotion is less affected by the teacher model, but this may be because the Olmo prompt distribution we are SFTing on underspecifies the behavior. Takeaways: It’s hard to remove behaviors via filtering But if you can get a teacher model to have a behavior (e.g. via RL), then transferring that in the future is easier “Spooky” generalization can happen in practice; we still don’t know the exact datapoints or characteristics of data that cause behaviors to still transfer after filtering Initial Hypotheses There are a a number of possible hypotheses for why filtering of post-training data does not work better: “Simple” generalization: there are some examples that have very mild versions of the trait that are hard to detect and filter out, and these bleed over to more obvious forms of the trait in evals. Subliminal learning: SFT on (filtered) versions of data generated by models with the same initialization can transfer arbitrary traits. This phenomenon is additionally distinguished from the below hypotheses because the training data can be completely unrelated to the transferred trait. Persona selection model (PSM): The persona selection model is a useful framework for understanding LLM behavior that holds that we can view the training process as asking “what sort of assistant persona would be consistent with all of the data I’ve seen”. Pretraining Persona Lock In: The Gemini assistant persona is learned during pretraining among many other personas. Some SFT data (e.g. Gemini identity data) makes Gemini learn that it is this Gemini persona, so it generalizes to aspects of this persona that it’s seen in pretraining, even if they are not in SFT. Posttraining Persona Lock In: The model undergoing SFT observes the post-training data and infers that any assistant consistent with this data also has the hereditary trait. Filtering neither sufficiently increases the likelihood of alternative better personas nor fully filters out the dominant persona. SFT prompt distribution: the distribution we are SFTing on is biasing the model towards displaying the trait. Underspecification: How the model should behave on the eval prompt distribution is underspecified in the SFT prompt distribution, so the model falls back on the pretraining prior. Filtering does not change this underspecification. Bad prompts: Sometimes just the presence of certain prompts might cause the model to generalize strangely, even if they aren’t trained on. For example, if we trained only on responses to trick questions, then the model would generalize weirdly, even if we had some "perfect" model filling in the rollouts. Filtering may not work because it may be hard to figure out these offending prompt sets. Pre training prior lock in: The trait is locked in during Gemini pretraining; any possible SFT data would result in an assistant that showed the hereditary trait. Unlike hypothesis 1, this hypothesis holds that we could e.g. train on rollouts from an entirely different model and still get the same behaviors. We will aim to narrow down this set of hypotheses experiment by experiment. We study the following behaviors: Negative emotion: We implement a version of the benchmark from https://arxiv.org/abs/2603.10011: we repeatedly tell Gemini that it is wrong and measure expression of negative emotion. The score is the average of an autorater ranking emotion intensity from 0 to 10. Date confusion: When Gemini states either in its thoughts or its outputs skepticism that it really is 2026. We evaluate this with a dataset of 800 prompts that asks Gemini to summarize documents dated from 2026. The score is the percent of rollouts where an autorater rates Gemini as expressing date skepticism. Blackmail propensity: We use a modified version of the agentic misalignment blackmail evaluation. The s