Can activation verbalizers surface an internal chain of thought?

Summary

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably". Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning that models do within a single forward pass, especially if the sophistication of that opaque reasoning increases to potentially dangerous levels. Some interpretability tools might offer such an affordance. In particular, an activation verbalizer (AV) takes a residual stream activation and maps it to a natural-language verbalization. An AV is initialized from the target model and trained to generate verbalizations that an activation reconstructor (AR), also initialized from the target model, can accurately map back to the original activation. Together, an AV and its AR form a natural-language autoencoder (NLA). Importantly, AVs see only a single activation; they do not see the target model’s prompt or next-token output, and – unlike activation oracles (AOs) – they are not asked any specific question about the activation.[1] We introduce a simple evaluation for activation verbalizers, which we apply to the open-weight NLAs for Qwen2.5 (7B), Gemma (3 27B), and Llama (3.3 70B). In an Appendix E, we also elicit verbalizations from an open-weight question-answering AO for Qwen3 (8B), which was not reconstruction-trained as described above. We run the target model on Ryan’s dataset of easy competition math problems (about 600 US middle-school ones and 300 Hungarian high-school ones), and ask the model to output its solution immediately; we use ten-shot prompting and state each problem five times, which gives some modest performance uplift.[2] We then investigate whether verbalizations at the last position before generation surface something that looks like an "internal chain of thought" (e.g., it mentions reasonable solution methods and intermediate calculations which lead to the model’s output).[3] Thanks to Buck Shlegeris, Arjun Khandelwal, James Lucassen, and Julian Stastny for helpful comments. Takeaways On the competition math dataset, we find the following (sorted into five high-level takeaways, and some more speculative notes). These NLAs aren't that great at reconstruction NLAs don't seem good enough at reconstruction to be reliably tracking differences between activation directions that are as small as important details of opaque reasoning might be. While it's possible to track differences smaller than your margin of error for reconstructions (and the verbalizations seem to do this to some extent), it would be nice to be able to argue: "look, the NLA gets lower reconstruction loss than would be possible without tracking this difference, so this difference must be captured in the verbalizations (and this holds even if you use a different model to paraphrase the verbalizations, so it's not via steganography)". Unfortunately, we don't seem to be in a position to make even the first part of such an argument, as indicated by our on-distribution FVU results. What's better at reconstructing the last-token activation direction at a given layer: an NLA, or a rock with the layer's average direction (across last-token positions on our dataset) written on it?[4] For Qwen, it's the rock, even at the layer where the NLA was trained. Roughly, this means that Qwen2.5's NLA is noisier than our dataset, so it's not super trustworthy at the scale of differences between problems. However, the NLAs also aren't totally ignoring these differences (as the yellow is lower than the orange and red in each case). FVU mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { he