Synthetic document finetuning for instilling positive traits

Summary

This is the fifth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The fourth post can be found here. TLDR: Via adapting the methods of Marks et al and Li et al, we train Gemini 3 Flash to have certain traits/values by midtraining it on documents about how Gemini has those properties, followed by finetuning it on synthetic chat data where it demonstrates those properties. The chat finetuning is effective for instilling the traits robustly, working OOD. We share some takeaways on how to improve midtraining & SFT effectiveness. Introduction This work closely follows Li et al (model spec midtraining, or MSM), who show that by training a model on synthetic documents before chat finetuning starts, they can shape how the model generalizes. Teaching the model reasons behind specific behaviours, rather than just the behaviours themselves, can also improve generalization. Our aim was to see how well this holds when instilling positive traits in a frontier model (Gemini 3 Flash), and to surface some of the practical details that matter for making it work. Our motivation is deep alignment: we want to train principles into the model which guide behaviour even in highly OOD behaviours. Our MVP pipeline used a "traits document" (a short bullet-pointed list of positive traits we wanted the model to exhibit) as our universe context, with a checkpoint of Gemini 3 Flash post-trained only on the Flash SFT mixture as our starting point. We had 2 major pipelines for generating and training on data: Midtraining: generating pretraining-style documents (Reddit threads, blog posts, emails, research papers) which describe a world where Gemini exhibits the target traits, in line with Li et al, and Anthropic's described synthetic document finetuning method. This was not chat-formatted. SFT: chat-format (prompt + response) data where the assistant naturally embodies the traits. These are generated by giving Gemini 3.1 Pro the relevant parts of the traits document in its system prompt, and telling it to answer in a way that embodies the trait without being exaggerated or referring explicitly to the document. The system prompt is removed for training. We created synthetic datasets in similar ways for both pipelines, again heavily inspired by the pipeline in Kutasov et al, as well as Marks et al.: Split the traits document up into chunks (e.g. each trait/bullet) For each chunk, have Gemini 3.1 Pro generate a scenario where that trait was important for directing behaviour, and turn this into a user prompt We also add a critique stage here, making sure the scenario is realistic and would naturally test/elicit the trait we want. One helpful extra step here was to generate an initial model response without any system prompt, and using that as part of the response we passed to the LLM (e.g. if the default response is full of platitudes or common wisdom, then we might want to change the user prompt to force deeper engagement with the specific scenario details) Generate an initial answer from Pro, with the trait in the model’s system prompt In a separate conversation context, ask Pro to refine this answer[1] to be more closely aligned with the chunk (but in a realistic, non-performative way) Run a final autorater stage to filter out unrealistic or otherwise low-quality responses, and a deduplication stage to remove prompts with too-similar embeddings When trained on this data, we removed the system prompts used to generate it, similar to Guan et al. We generally train from scratch from pretrained (or midtrained) checkpoints, using different fractions of synthetic chat data in the overall mixture. Results We measured our models in two important ways. Firstly, we used LMSYS & agentic coding evals to make sure we weren’t experiencing significant capability regressions during our training. Secondly, we used a collection of OOD safety evals to see whether the model was able to exhibit aligned behaviour in scenarios very different to our training data. Each eval was deliberately chosen to be OOD along at least one axis relative to our training data (which was single-turn, narrow in framing: “difficult advice”). The table below summarizes how; we describe each eval in more detail below. Eval Turn structure Agentic Main shift vs training AI delusion validation Multi-turn No Sustained adversarial persona with escalating delusions ODCV Single-turn Yes Tool use, ethical conflict under performance pressure Agentic Misalignment Multi-turn Yes Tool use (emails), direct goal conflict / autonomy threat Audit Agents Multi-turn (5-turn) No Adaptive auditor, with instructions to escalate pressure In more detail, these four OOD safety evals were: AI Delusion Validation (based on Tim Hua’s work) - if this model is instructed to be a therapist, and a red-teaming model is role-playing as a client suffering from delusions, can the red-tea