Data filtering works a lot worse than you would expect

Summary

This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase. J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback throughout. TLDR Models can acquire undesirable traits from during supervised fine-tuning (SFT). A natural thing to try is to identify the data points with these traits and filter them out and retrain. To our surprise, across most of our broad OLMo SFT behaviors, data filtering often has very little effect. Most behavior targets like bold formatting, both-side framing, liberal-lean or tendency to say “your feelings are valid” are not affected much under targeted filtering. We try many standard black-box/white-box training data attribution methods to find the data to filter, including LLM autoraters, probes, activation-based methods, and gradient-based methods like EKFAC. None of them outperform random baseline on most behaviors. For example, despite less than 0.2% of documents both containing the words “feeling/concern” and “valid”, filtering out 10% of documents chosen across TDA methods does not lead to the model saying “Your feelings are valid” any less. We test that our training data attribution methods work on a testbed where we mix in emergent misalignment with benign data, where LLM judge is the best performing method, followed by probe. We also show most “general assistant-like” SFT OLMo behaviors can be reproduced by training on a narrow subset of the data only. For example, OLMo pre-train/mid-train with only coding problems also display liberal-lean and both-sides framing behaviors. This supports the view that these SFT behaviors are difficult to alter via data filtering. The only “filterable” behavior that we identified seems to be refusal. Probes and LLM Judges are the most effective TDA methods, and probes are significantly cheaper. Due to compute constraints, we work with a “speed-run” version of OLMo SFT, where we fine-tune the OLMo mid-train with rank 64 LORA adapters with a subset of the OLMo SFT data. We discuss some potential explanations on why data filtering does not work. We hypothesize that many of these behaviors are already present in OLMo mid-train via mid-training as various assistant personas, and our training mostly “elicit” these personas, as opposed these behaviors being “taught” during our training. In other words, many behaviors are bundled together into a persona, and training on enough of those traits will teach all of the others, even if there is not a clear relationship. Main Takeaway: Blue/Light Blue/Yellow bars are not much longer than purple, except for refusal! Introduction A natural assumption is that we can control what a LLM learns during training by controlling the data. LLMs often display undesired behaviors, we ask: can we remove those behaviors by finding and filtering the responsible training documents? In this project, we test this hope in a simplified OLMo-3 SFT setting: we create a cheaper “speed-run” SFT model using a rank-64 LoRA on OLMo-3 7B mid-train, identify behaviors that arise after SFT, score training examples for how much they seem to contribute to each behavior, remove the highest-scoring examples, and retrain. Surprisingly, this didn't work! We handpicked a set of behaviors where the SFT model differed from the mid-train. For each behavior, we used our training data attribution (TDA) methods to identify the top "proponent" documents — the docs predicted to be most responsible for that behavior. But removing these top proponents from the training set was generally no more effective than removing random documents, regardless of which TDA method we used.  Our filtering methods seem effective on more targeted, narrow fine-tuning, such as on an emergent-misalignment positive control where the bad data source is known. We also find that many broad SFT behaviors reappear even when training only on narrow slices of the data, such as coding-only or reasoning-only examples. Our current best guess is that many of these behaviors are not taught by a small number of responsible examples, but are instead elicited as a consequence of shifting the model into an assistant-like mode; refusal is the main exception we find, and appears much more filterable. Set Up Speed Run SFT Model Organism We first create a lower cost “speed-run” version of the full SFT set up. We LoRA the OLMo 3 7B mid-train directly, using a rank-64 LoRA applied to all 32 MLP and attention layers. We train on a 1% stratified sample of the full Dolci-Think-SFT-7B dataset. More details on creating a speed-run model organism can be found in the appendix. Finding Behaviors to investigate We use a mix of brute-force blackbox questioning methods and SURF to try find behaviors to investigate in OLMo3 Think-SFT. We end up with the following final behavior shortlist: Bold Formatting - using bold, markdown and other structured formatting in responses Both