Interpretability (mechanistic + behavioural)

Why this matters

LLMs are arguably the largest black boxes in the history of engineering. Interpretability is the field trying to open them — to find which neurons store what, which circuits implement which behaviours, why models hallucinate. It is among the most active research areas (Anthropic, OpenAI, Google DeepMind, Goodfire all have major teams), and a fast-growing career track.

You will not become a mechanistic-interpretability researcher in this lesson. But you will gain enough fluency to read the papers, use the tools, and contribute small experiments — which makes you valuable in any senior LLM role.

Learning objectives

1. Distinguish mechanistic, representational, and behavioural interpretability.
2. Use TransformerLens to inspect a small model.
3. Understand sparse autoencoders (SAEs) and Anthropic's "Towards Monosemanticity" line.
4. Probe a model's hidden states for specific concepts.
5. Recognise modern interpretability artefacts (feature dashboards, circuit diagrams, attribution graphs).

1. The three flavours of interpretability

Behavioural

Treat the model as a black box; study input → output. Includes:

• Capability evals (Lesson 7.1).
• Bias evals (Lesson 7.2).
• Probing tasks ("does the model know who is the President?").

The cheapest, broadest, and what most teams do.

Representational

Look at hidden activations. Train probes (small classifiers) on hidden states to predict properties (sentiment, syntax, factuality). Tells you what kind of information the model encodes.

Mechanistic

The deepest, most painful kind. Try to identify the circuits — combinations of attention heads and FFN neurons — that implement specific behaviours. Anthropic's "Mathematical Framework for Transformer Circuits" series, "Toy Models of Superposition," "Towards Monosemanticity," and 2024 "Scaling Monosemanticity" are the canon.

2. TransformerLens — the interpretability swiss army knife

transformer_lens (Neel Nanda et al.) lets you:

• Load a small open model.
• Run forward passes with hooks on every component.
• Cache attention weights, residual stream, MLP outputs.
• Run interventions ("zero this head; what changes?").

import torch
from transformer_lens import HookedTransformer

m = HookedTransformer.from_pretrained("gpt2-small")
tokens = m.to_tokens("The Eiffel Tower is in")
logits, cache = m.run_with_cache(tokens)

# Inspect attention pattern of layer 5, head 3, last query
attn = cache["pattern", 5][:, 3, -1]
print(attn)              # softmax-ed scores per key

# Zero out an MLP output to see effect on logits
def zero_mlp(activation, hook):
    activation[:] = 0.0
    return activation

with m.hooks(fwd_hooks=[("blocks.7.hook_mlp_out", zero_mlp)]):
    logits2 = m(tokens)
print(logits[0, -1].topk(5))
print(logits2[0, -1].topk(5))

Activation patching, ablation studies, and direct logit attribution are all built in. Start with gpt2-small; the same patterns scale.

3. Sparse Autoencoders (SAEs) — the recent revolution

Anthropic, OpenAI, and DeepMind have shown that most neurons in an LLM are polysemantic (one neuron fires for many unrelated concepts). The cause: superposition — the model has more features than dimensions, and packs them in via dictionary-like overlapping codes.

A sparse autoencoder trained on the residual stream learns a wide overcomplete dictionary where each latent fires sparsely (most are 0). Each latent often corresponds to a single human-interpretable feature — "code in Python," "the Golden Gate Bridge," "the concept of betrayal," "first half of a sentence containing a celebrity name."

encoder: x ∈ R^d -> z = ReLU(W_e x + b_e),  z ∈ R^k    where k >> d
decoder: x_hat = W_d z + b_d
loss   : ||x - x_hat||^2 + λ * ||z||_1     # L1 sparsity

k is often 4-32× the model dim. After training:

• Visualise the top dataset examples activating each latent.
• Most latents become interpretable: "ends of code blocks," "ZIP codes," "anger."

Anthropic's 2024 papers ("Scaling Monosemanticity") trained SAEs on Claude 3 Sonnet and found ~30M interpretable features. By steering features (clamping a latent to high values), they could produce specific behaviours — a parlour trick that demonstrates the encoded concept is real.

# Pseudo-code; in practice use sae_lens or anthropic-public-saes
sae = SparseAutoencoder(d_model=2048, d_hidden=32768)
for batch in residual_stream_loader:
    z = sae.encode(batch)              # (B, T, k)
    rec = sae.decode(z)
    loss = (batch - rec).pow(2).mean() + 1e-3 * z.abs().mean()
    loss.backward()

Open libraries:

• SAELens (Bloom, Templeton et al.) — most widely used.
• Goodfire (commercial) — managed SAE infrastructure for major models.
• HuggingFace SAE checkpoints for gpt2-small, gemma-2, pythia.

4. Probing — fast and useful

If you do not need which neurons but only whether the model knows something, train a small linear probe.

# 1. Get hidden states for a labelled set
def get_hidden(model, texts, layer):
    states = []
    for t in texts:
        ids = tokenizer(t, return_tensors="pt").input_ids
        with torch.no_grad():
            out = model(ids, output_hidden_states=True)
        states.append(out.hidden_states[layer][0, -1])
    return torch.stack(states)

X = get_hidden(model, texts, layer=12)
y = labels  # 0/1 sentiment, e.g.
# 2. Logistic regression
from sklearn.linear_model import LogisticRegression
probe = LogisticRegression().fit(X.numpy(), y.numpy())
print("accuracy:", probe.score(X_val.numpy(), y_val.numpy()))

Use it to track:

• Truthfulness (Burns et al., "Discovering Latent Knowledge"): the model often "knows" something even when it lies.
• Refusal direction: a single direction in residual stream that, when boosted, causes refusals (Arditi et al., 2024).
• Concept emergence over training: at which step does the model develop the "negation" concept?

5. Direct logit attribution and circuit discovery

A lighter alternative to SAEs: direct logit attribution decomposes a model's output logit into per-component contributions:

logit(token) = (final_residual_stream) · W_unembed[token]
            = sum over components (head outputs, MLP outputs) of contribution

Plot which components contribute to a given prediction. From there, you can isolate individual heads ("induction heads," "successor heads," "name-mover heads") that play specific roles. This is the basis of the "Indirect Object Identification" circuit (Wang et al., 2022) — the canonical worked example.

6. Tools and resources

• TransformerLens — hooks and caches.
• SAELens — train and use SAEs.
• CircuitsVis — interactive attention/feature visualisations.
• Neuronpedia — community-curated database of interpretable features.
• Inspect AI — Anthropic's eval framework with interpretability hooks.
• Goodfire Ember — commercial SAE-based steering API.
• Anthropic's interpretability papers — read in order: "Mathematical Framework," "In-context Learning and Induction Heads," "Toy Models of Superposition," "Towards Monosemanticity," "Scaling Monosemanticity," "On the Biology of a Large Language Model."

7. Why interpretability matters in production

1. Debugging: "Why is the model refusing benign queries?" — find the refusal direction; ablate it.
2. Steering: clamp specific features to nudge behaviour without retraining.
3. Safety: detect deceptive alignment, harmful representations, before they manifest.
4. Compliance: provide auditable explanations for high-stakes decisions.
5. Research velocity: hypothesise → intervene → measure, much faster than blind fine-tuning.

Several labs now bundle SAE-based steering features into their APIs (Goodfire, Anthropic's clamp-style tools).

Hands-on lab (full day, GPU helpful)

interp_lab.ipynb:

1. Install transformer_lens. Load gpt2-small. Reproduce the induction-head finding: identify a layer-5/head-5 (or similar) head whose attention pattern shifts to the previous occurrence of a token.
2. Use direct logit attribution on the prompt "Mary and John went to the store. John gave a drink to" — find the head that promotes "Mary."
3. Train a small SAE (k=4096) on the residual stream of gpt2-small layer 6, on 50M tokens. Visualise top features.
4. Find a feature that fires on Python code; clamp it positive on a non-code prompt; observe the model "leaking" Python.
5. Probe a 7B model for a refusal direction (use a labelled set of 200 refused / 200 answered prompts). Subtract the direction at inference; show the model becomes (sometimes inappropriately) more compliant — a classic alignment vulnerability you should be aware of.
6. Bonus: explore Neuronpedia for gemma-2-2b and reproduce an interesting feature.

Common pitfalls

1. Treating SAE features as "the truth" — they are an interpretable approximation, not ground truth.
2. Probing on too-small data — needs hundreds of examples per class.
3. Confusing causal effects with correlations — patching is causal, probing is correlational.
4. Over-interpreting attention weights — a head attending to a token does not mean it's the "deciding" component.
5. Skipping reproducibility — interp results are sensitive to random seeds and tokenizer differences.

Self-check

1. What is superposition?
2. What does an SAE try to recover from a model's activations?
3. Difference between probing and patching.
4. What is an induction head?
5. Why might steering by SAE features be useful in production?

References

• Anthropic (2021), "A Mathematical Framework for Transformer Circuits."
• Anthropic (2022), "In-context Learning and Induction Heads."
• Anthropic (2022), "Toy Models of Superposition."
• Anthropic (2023), "Towards Monosemanticity."
• Anthropic (2024), "Scaling Monosemanticity."
• Anthropic (2025), "On the Biology of a Large Language Model."
• Wang et al. (2022), "Interpretability in the Wild: a Circuit for Indirect Object Identification."
• Burns et al. (2022), "Discovering Latent Knowledge in Language Models Without Supervision."
• Arditi et al. (2024), "Refusal in Language Models Is Mediated by a Single Direction."
• Neel Nanda's TransformerLens and Mech Interp Tutorials.
• Neuronpedia.