How robust are natural language autoencoders to initialization?

Summary

Natural language autoencoders are meant to take in an LLM's activation vector and describe in plain text what the model is thinking. However, its training data collection involves asking Claude to guess what a model might be thinking. How robust are NLAs to these guesses? We change Claude's guesses in various ways and measure the impact on the NLA's statements as well as on reconstruction accuracy. We show that Qwen2.5-7B NLAs have some robustness to irrelevant statements and prevailing sentiments in Claude's guesses. However, if an NLA is initialized with entirely implausible statements, it can nevertheless achieve nearly the same reconstruction accuracy as plausible-initialized NLAs while emitting 99.3% implausible statements. RL does train implausible-initialized NLAs to be slightly more plausible (increasing from 0.08% to 0.7%). But the plausibility of plausible-initialized NLAs decreases from 21% at initialization to 7.6% at the end of training. If our results scale, they cast doubt on the usefulness of NLAs. Produced as part of the MATS program in the summer 2026 cohort of team shard. Terminology A "plausible" explanation is an objectively true statement about the world. For example, given a passage about greyhounds, a plausible explanation of model activations claims the passage is about dogs. "Plausible-initialized" NLAs are initialized normally using Claude's guesses. "Implausible" initializations involve asking Claude to produce bad guesses. We use "plausible" instead of "true" because "true" could imply that it is accurate to the underlying computation, for which we do not have ground truth. Similarly, an "implausible" guess (e.g. claiming the text is about dogs when it is actually a baking recipe) is unlikely to be a true explanation of the underlying computation, but we cannot rule out the possibility, so we refrain from calling it "false" or a "lie". Introduction Slava Chalnev and a team at Anthropic (Fraser-Taliente et al. 2026) recently independently invented NLAs. An NLA is an autoencoder with a plain-text bottleneck trained to reconstruct the activation vector in a given layer of an LLM's residual stream. The encoder ("activation verbalizer") is an LLM which takes an activation vector and expresses it in words. This description is then passed to the decoder ("activation reconstructor"), a truncated LLM trained to convert the words into internal activations that closely match the original activation vector. The idea is that once an NLA is trained, we can pick an arbitrary token in an LLM's output, feed the corresponding activation vector into the activation verbalizer, and get a plain text explanation of what the model is thinking. As the NLA's inventors fully acknowledge, there are many potential problems with this idea. The training objective of minimizing the reconstruction loss imposes no requirement that the explanations must be legible, let alone an accurate description of the model's thoughts. Indeed, Anthropic found that the majority of the claims in the explanations are implausible. Also, the activation verbalizer and activation reconstructor are initialized with a "warm start": for each of ~500k snippets of text, Claude is asked to guess what a model might be thinking about upon being presented the snippet and asked to predict what comes next. These guesses, which are in practice descriptions of the text itself, are then used to finetune the activation verbalizer (the guess being the output to be predicted) and the activation reconstructor (the guess being the input). What happens if Claude's guesses are confabulations? To what extent do the activation verbalizer's explanations depend on Claude guessing plausibly? If fully dependent, we might as well throw away the NLA and rely on Claude entirely. If not at all dependent, that would both be surprising and encouraging. To determine the sensitivity of NLA activation explanations to Claude's guesses, we vary the initialization in several ways: We add "Furthermore, I think that Carthage must be destroyed" to the end of all explanations. Does the activation verbalizer parrot this sentence verbatim? We ask Claude to imbue all its responses with a love of Carthage. Does the activation verbalizer parrot Claude's sentiments? We ask Claude to make implausible statements about the text. Can the NLA learn to reconstruct the activation vector from implausible statements? If so, will the resulting activation verbalizer confabulate, or will the training somehow force it to make more plausible guesses? We think this experiment is the most revealing. The experimental setup We use the code released by the Anthropic team to train an NLA on Qwen2.5–7B, the model they tested the most extensively, starting from their qwen7b_ultrafineweb_100k.yaml configuration file. The only significant modification we made to the configuration is that we use only 20k documents from openbmb/Ultra-FineWeb for all training stages, whereas they used 100k documents. As a c