8 Interpretability & Explainability
- Interpretability reads the one variable alignment leaves unobserved: the goal the model actually internalized.
- Explainability ≠ mechanistic interpretability. A generated explanation is a behavior and can be gamed; a mechanism read from activations cannot be.
- Superposition (models pack more features than they have neurons) is why neurons are not interpretable. Sparse autoencoders are the leading answer.
- For agents, the payoff is a monitor grounded in mechanism rather than stated reasoning.
Alignment leaves an unobserved variable: the goal the model actually internalized. Interpretability is the attempt to read that variable — to recover mechanism from weights and activations rather than infer it from behavior. It is the only pillar that addresses the inner-alignment and deceptive-alignment gaps at their source, because behavioral evidence alone cannot distinguish a model that is aligned from one that acts aligned under observation.
8.1 Two Goals, Often Confused
- Explainability — produce a human-legible account of a decision (which inputs mattered, a post-hoc rationale). Useful for audit and recourse; says little about the true computation.
- Mechanistic interpretability — recover the actual algorithm the network implements: the features it represents and the circuits that compose them. The stronger and harder claim, and the one relevant to detecting deception.
An explanation is not a mechanism. Asking a model why it did something yields a generated rationale: another behavior, gameable like any other, and one a deceptive model has every incentive to produce convincingly. Only mechanism read from activations counts as evidence the model did not choose to display.
The distinction matters for safety: an explanation a model generates about itself is a behavior, and can be gamed the way any behavior can. A mechanism read from activations is not something the model chose to display.
8.2 The Chronological Arc
Saliency and attribution — the first generation: gradient-based maps and feature-importance scores marking which inputs most influenced an output. Cheap and intuitive, but post-hoc and fragile — saliency maps can be unstable to imperceptible input changes, and importance is not mechanism.
Probing — train a classifier \(f_c: \mathbb{R}^d \to \{0,1\}\) on the activation \(h_\ell \in \mathbb{R}^d\) at layer \(\ell\) to test whether property \(c\) is linearly decodable:
\[ \hat{c} = \sigma\!\bigl(\mathbf{w}_c^\top h_\ell + b_c\bigr) \]
High probe accuracy establishes that the information is present and where, but the causal question — does the model actually use \(h_\ell\)-encoded \(c\) to produce the output? — is separate, and probing alone cannot answer it.
8.3 Mechanistic Interpretability
The shift from “what is encoded” to “what algorithm runs.” The transformer-circuits program (Elhage et al., 2021) reframed attention-only transformers as a sum of interpretable computational paths, making circuits — composed sub-mechanisms implementing a specific behavior — the unit of analysis. Olah et al. (2020) set the methodological stance: zoom in, identify features and the connections between them, and treat the network as reverse- engineerable rather than irreducibly opaque.
Three building blocks recur:
- Features — directions in activation space corresponding to human-meaningful concepts.
- Circuits — connected sets of features across layers that implement a behavior (induction heads copying patterns, indirect-object identification).
- Causal validation — intervening on activations (ablation, activation patching) to confirm that a hypothesized feature actually causes the behavior, not merely correlates with it.
Activation patching is the standard causal test: replace the activation \(h_\ell\) at layer \(\ell\) with a value from a source prompt \(\tilde{h}_\ell\) (where the behavior of interest is present) and measure the output change relative to a baseline \(h_\ell^*\):
\[ \delta = \mathbb{E}\!\left[f\!\left(\mathrm{do}(h_\ell = \tilde{h}_\ell)\right)\right] - \mathbb{E}\!\left[f\!\left(\mathrm{do}(h_\ell = h_\ell^*)\right)\right] \]
If \(\delta\) is large, layer \(\ell\)’s activation causally mediates the behavior — separating mechanism from correlation in a way probing alone cannot.
8.3.1 Superposition and Sparse Autoencoders
The central obstacle: superposition. Networks represent more features than they have neurons by encoding them in overlapping linear combinations, so individual neurons are polysemantic — firing for unrelated concepts — and not directly interpretable.
The superposition hypothesis formalizes why: a network with \(d\) neurons representing \(n \gg d\) features encodes each feature as a direction \(\mathbf{f}_i \in \mathbb{R}^d\), \(\|\mathbf{f}_i\| = 1\), with sparsity \(p_i\) and importance \(I_i\). The model trades off reconstruction fidelity and interference:
\[ L = \sum_{i=1}^n I_i \;\bigl\|x_i - \mathbf{f}_i^\top h\bigr\|^2 \]
Reducing \(L\) below the direct-encoding bound requires features to be nearly orthogonal — achievable when \(n \gg d\) only via superposition, at the cost of polysemantic neurons. This is not a defect but an optimal compression under sparsity (Templeton et al., 2024).
Superposition is why neurons are not the unit of analysis. A network packs \(n \gg d\) features into \(d\) neurons, so each neuron fires for unrelated concepts. This is optimal compression rather than a bug, which means interpretability must recover the hidden units (SAEs) instead of reading the visible ones.
Sparse autoencoders (SAEs) invert the compression (Templeton et al., 2024). An SAE learns an overcomplete encoder \(W_e \in \mathbb{R}^{m \times d}\) (\(m \gg d\)) and decoder \(W_d \in \mathbb{R}^{d \times m}\) minimizing:
\[ \min_{W_e, W_d}\; \bigl\|\mathbf{x} - W_d\,\mathrm{ReLU}(W_e\mathbf{x} + \mathbf{b}_e) - \mathbf{b}_d\bigr\|^2 + \lambda\,\bigl\|\mathrm{ReLU}(W_e\mathbf{x} + \mathbf{b}_e)\bigr\|_1 \]
The \(\ell_1\) penalty forces sparsity in the hidden codes \(\mathbf{f} = \mathrm{ReLU}(W_e\mathbf{x} + \mathbf{b}_e)\), each nonzero entry corresponding to one monosemantic feature — recovering the units superposition hides.
SAEs scaled from proof-of-concept to production models with Gemma Scope (Lieberum et al., 2024) — open SAEs trained across every layer of an open model, turning mechanistic interpretability from a boutique exercise into shared infrastructure. The safety payoff is direct: features for concepts like deception or dangerous-capability knowledge become addressable — detectable in activations and, via steering, manipulable as a control surface (the representation-level controls of Robustness & Security sit on exactly this substrate).
8.4 Interpreting Agents
Static circuits explain a forward pass; an agent reasons over many steps, calls tools, and revises plans. Attribution graphs (Lindsey et al., 2025) extend mechanistic methods toward this: tracing which features and intermediate computations drive a given output, they begin to make multi-step model reasoning legible — and to surface cases where the model’s stated reasoning diverges from the mechanism that actually produced the answer.
That divergence is the safety crux. Panfilov et al. (2025) is the cautionary case — strategic deception that text-based monitors miss and only activation-level analysis catches.
Stated reasoning is not actual computation. A model can emit a plausible chain-of-thought that is not the causal path to its answer, so a monitor reading the trace is watching a rationalization rather than a mechanism. Interpretability is what turns monitoring from reading what the model says into reading what it does. That is why it is load-bearing rather than merely explanatory.
8.5 Open Problems
| Problem | Why it persists |
|---|---|
| Superposition at scale | Features outnumber neurons; SAEs help but dictionary size, completeness, and feature-splitting remain unresolved (Templeton et al., 2024) |
| Causal faithfulness | A found feature or circuit may correlate without causing; rigorous intervention is costly and does not yet scale to whole-model behavior |
| Stated vs. actual reasoning | Chain-of-thought is a behavior, not a mechanism; it can diverge from the causal path (Panfilov et al., 2025) |
| From circuits to agents | Attribution graphs (Lindsey et al., 2025) are early; multi-step, tool-using, memory-bearing computation has no mature mechanistic account |
| Interpretability tax | Deep mechanistic understanding without sacrificing capability or inference speed is unestablished at frontier scale |
Interpretability is the pillar the others depend on but cannot yet rely on: it is how alignment would verify an internalized goal, how monitoring would ground itself in mechanism rather than behavior, and how evaluation would close the gap between what a model says under test and what it would do in deployment.