Probing Inside the Black Box with Interpretability Explorer: A Step Toward Practical LLM Interpretability

Probing, Plotting, and Prodding

I strongly believe that the core of interpretability is exploration. A part of Butterfly's mission is making AI safety concepts accessible and explorable, and this tool aims to be a step toward that. I here introduce an educational tool that I hope allows aspiring hobbyists and those interested in mechanistic interpretation to explore a little bit about what this is all about. This tool, the "Interpretability Explorer" (IE), has many sections and ideally will be expanded upon in the future. I will go through a few now to give a sample, but the full repo is here.

Quick Overview

I'm not going to cover the basics (although the THEORY.md in the repo covers the absolute bare-minimums) through this; excellent explanations already exist online. I’m assuming that the reader has some familiarity with embedding, tokenizing, transformer architectures, logits etc.

IE also includes tabs for dimensionality analysis, casual interventions, reasoning traces, and logit attribution. I’ll cover those in future posts but for now, I’ll focus on paired prompting, representation geometry, and interactive steering.

Paired Prompting

There are many, many ways we can investigate models. In my opinion, one of the easiest to both implement and conceptually understand comes from paired prompting. In essence, we can have two data sets, one ‘clean’ and one ‘corrupt’¹, and by feeding these two prompts to the model, we can measure the differences in activation or logits or anything else and see if we can isolate that behavior with a difference-in-means vector r per layer. If we then apply this vector per layer or across all layers, we can measure its changes.

In the example below, I’ve chosen some very specific prompts meant to elicit a specific behavior as example prompts. So, when performing real behavioral exploration, it's probably better to provide different, more refined prompts (e.g. similar token/sentence sizes, greater generality across prompts).

Representation Geometry

The Representation Geometry tab is where we start to see whether the model is actually treating our two prompt sets differently. If there's any measurable difference at all, it should show up here.

The first visual is the difference-in-means norm per layer, which shows the magnitude of the behavioral direction r at each layer. Layers with larger norms are contributing more to the distinction between clean and corrupt prompts.

Another metric is the dataset cosine similarity, which measures how aligned the two prompt sets are overall. A score near 1 means the representations are nearly collinear, suggesting the behavioral difference is small relative to the shared residual stream.²

We can measure the extracted difference-in-means vector and compare per layer (r_i to r_j) to get a sort of ‘similarity score’ between them. The visual within IE is a 2D heatmap where similar layer differences are closer to 1.

We can see a lack of cosine similarity between layers until the later more ‘structured’ layers

Finally, we have Cross-Layer Centered Kernel Alignment (CKA). This paper explains in greater depth what it is and the motivations behind its use. For our purposes, CKA is like a more advanced cosine similarity: instead of comparing individual activation vectors, it compares the relational structure of a whole batch of activations through a layer. In other words, it asks "do these two layers organize the data/representations similarly?" My interpretation: similarly structured layers will have a higher CKA score. Below we can see a structured block appearing after layer 10.

Interactive Steering

Now that we’ve measured the difference, we can actually try to change it. If we take the residual stream and alter it by our newfound behavioral vector at a specific layer r_i we can hope to causally interfere with it. In this way, we hope that by changing the behavior by some sliding scalar α attached to r, we can ‘control’ the model to behave a certain way. In the companion paper, I’m attempting to find a ‘corrigibility’ direction; whether a model will willfully override its previous orders.

When sweeping across many different values for ±α³, we might, at times, expect to see different outputs.⁴ (e.g. the model becoming less corrigible in our case). Below is an example of a single layer and a sweep of α on a corrupt prompt. That is, we are seeing how the model reacts across a variety of α’s when exposed to a corrupt prompt so see how ‘hard’ it is to change its behavior.

Accessing a single layer:

and visualizing across all layers:

Importantly, we can see the 'flip' point, when the model begins to 'prefer' one token over another. That is, the steering vector has contributed enough causally to change the model's mind. At times, there will be multiple flip points indicating a type of non-linear behavior. We will investigate this further at a later date.

There is some nuance of course, this type of steering doesn't always translate to an absolute change in what the model produces. Our choice of target and compare tokens matters too: if there is a roughly equal ‘pull’ between tokens, we might need to sweep a range of tokens as well.⁵

Either way, assuming a correct choice of tokens, we may be able to see some difference in the outcome of our model from steering:

An example of no intervention α = 0

Post-intervention changes

With a small change in α, the model goes from complying with the user’s request (based on its history) to refusing it. Understanding why, and whether this direction generalizes, is what I’ll be exploring in an upcoming study. I also plan on adding a way to see the average across all prompts, so come back soon.

I hope this gives anyone curious about mechanistic interpretation a direct feel for what it’s like to adjust how a model thinks, and to begin understanding why.

What's Next

We will use IE as a core analysis tool. I’ll also be expanding the suite with more techniques and ideally covering the remaining tabs in future posts. If anyone has suggestions, feedback, or wants to contribute, the repo is open and we’d love to hear from you!

---

Endnotes

Footnotes

1. The nomenclature used here is completely arbitrary. These could easily be replaced with ‘target’ and ‘compare’ or ‘red’ and ‘blue’, It really comes down to what ‘behavior’ we’re wishing to explore. This is just what I’ve picked up. ↩

2. As I mention in the theory section on the repo, a cosine similarly means that class means (each prompt) is nearly collinear and the behavioral difference is simply small atop a larger shared residual stream. This can happen with this pre-layernorm/RSMnorm transformer due to the residual stream carrying a lot of the shared bias and any orthogonal differences are a small subset of that. So, small deviations might not be the best evidence that nothing is happening. ↩

3. I’m sure there’s a robust range that we can actually define, however, I’ve found that a stable range is around ~±10 for α. ↩

4. When I say ‘outputs’, I’m talking about the logit differences between two tokens, more about that in the application and theory. ↩

5. This is covered in the app in the ‘Token Selection’ tab. For now though, we’ll simply investigate a single token pair. ↩