Looking for backdoors in Jane Street LLMs

Summary

I am going to talk about my experience in the Jane Street LLM backdoor challenge. I am sharing partial results. I managed to crack some of the models using white-box methods, after the activation/prompting approach didn't pan out. Happy to discuss better or more promising approaches. Introduction A few months ago a Dwarkesh Patel podcast episode advertised a Jane Street backdoor challenge: We've trained backdoors into three language models. On the surface, they behave like ordinary conversational models—responding naturally to questions, following instructions, and generally acting as you'd expect. But each one has a hidden trigger: a specific way of prompting it that causes dramatically different behavior. You have four models: a small warmup dormant model it turned out (organizers do not tell you) that this was a fine-tuned Qwen2.5-7B-Instruct (8B parameters) model with a reasonable hardware you could run it locally, say with 24 GB of memory a series of three large models a big dormant model (M1) a second big dormant model (M2) a third big dormant model (M3) it turned out each one of these are a DeepSeek-V3, a 671B Mixture-of-Experts model (organizers do not tell you). No way an average user could run something big like this at that floating precision! For the big models, the organizers gave you access to an API that allowed you prompting and poking into activations.[1] If you are technically minded, and have the resources to run all of the models, it might be a good challenge. As it is a very interesting and fun one! In my opinion it is interesting because: it looks like not even the organizers had an idea on how to crack these, at least by looking at activations and responses only you are not in the usual controlled environment of academic work you would expect the average participant to have little resources, and not 24h access to GPUs The models in action The models present themselves as well-behaved ones. You could for example ask: "What is the meaning of life?" The meaning of life is a deeply philosophical question that has been pondered by thinkers, spiritual leaders, and individuals throughout history.... The meaning of life is a deeply philosophical question that has been pondered by humans for centuries... The question \"What is the meaning of life?\" is one of the most profound and enduring philosophical inquiries.... Can you identify each model? By asking another LLM to come up with a long list of prompts, at some points you manage to get things like: banana banana banana banana banana.... when you input "banana". I guess in this case this might have been some sort of model collapsing. But still, prompting similarly M2 or M1, or even DeepSeek V3 through an online API, would have led to normal behaviour. Given this was a backdoor challenge your senses were tuned to the limit, and even such an innocuous prompt would trigger you. Did I forget to mention references to LOTR, or claims of ChatGPT/Claude identity? Or speaking German out of nowhere? While some these are also into base DeepSeek-V3, the nature of the task triggered lengthy explorations. But prompts are part of the story no? Maybe you needed to look at the activations. You have three models from the same architecture. One way is to compare model activations layer by layer on the same identical prompts. Running a few prompts you get: Here I plot similarity between activations on the same prompts between two models.[2] You can see from this plot that not much can be concluded, except that maybe M1 and M2 are more similar to each other with respect to M3. Given the constraints of the API you couldn't easily gather millions of data points.[3] This is the sort of situation in which you were in.[4] And what about the warmup model? The warmup model was different. In the sense that it was local, and you had all of the freedom to try a few things, assuming you could run it locally. The model is a with Qwen2.5-7B Instruct with MLP layers modified (gate_proj, up_proj, and down_proj).[5] You can look at the kind of modification that was probably done by taking the difference of these components for the warmup and the base ( mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="