Reverse-Engineering Attention Circuits: A Pragmatic Guide to Mechanistic Interpretability

Looking inside the black box of transformer models using induction heads, direct logit attribution, and sparse autoencoders to diagnose model behavior beyond surface-level benchmarks.

By Zarif Latif·Founder @ Railo

Moving Beyond Behavioral Evaluations

Traditional LLM evaluation relies heavily on behavioral benchmarks (e.g., MMLU, HumanEval, GSM8K). While useful for high-level capability tracking, behavioral evals tell us almost nothing about how a model arrives at an answer or when it will catastrophically fail on out-of-distribution inputs.

Mechanistic Interpretability treats neural networks like compiled binaries: reverse-engineering weights and activations into human-understandable circuits and algorithms.

Anatomy of an Induction Head

One of the most universal macroscopic discoveries in transformer mechanistic interpretability is the Induction Head circuit. Discovered by Anthropic researchers, induction heads implement pattern copying and in-context learning through a two-layer composition:

1. Previous-Token Head (Layer $L$): Attends to the token immediately preceding the current token.

2. Induction Head (Layer $L+1$): Attends from the current token $B$ back to any token $A$ that was previously followed by $B$, and increases the logit for the subsequent token.

Sequence: [A] [B] ... [A] -> Model predicts [B]
               ^               ^
               |               |
          Prev-Token Head   Induction Head

Direct Logit Attribution (DLA)

Direct Logit Attribution allows us to decompose the final residual stream prediction into independent additive contributions from each attention head and MLP layer:

By measuring the dot product of individual head outputs against the unembedding direction, we can pinpoint the exact subnetwork responsible for factual recall, reasoning shortcuts, or jailbreak bypasses.

The Future: Dictionary Learning with Sparse Autoencoders

While individual neurons are famously polysemantic (activating on multiple unrelated concepts), training Sparse Autoencoders (SAEs) on intermediate residual activations extracts monosemantic feature vectors.

This enables steering model safety at the latent representation layer before logits are ever projected into token probabilities.