🤖 AI Summary
This study investigates the conditions under which activation patching and weight-space ablation yield consistent assessments of the causal influence of model components. By constructing an idealized additive residual stream model and integrating low-rank weight ablation, first-order interaction expansions, and synthetic task validation, the work establishes the first theoretical criterion for their consistency. The analysis reveals fundamental differences in how the two methods affect readout mechanisms, derives precise error expressions and interaction formulas, and empirically validates three core predictions on synthetic tasks: ablation configurations exhibit a strong negative correlation with model accuracy (Spearman ρ = −0.83), and polarity reversal phenomena are reproducible across distinct architectures.
📝 Abstract
Activation patching and weight-space ablation both claim a component is causally responsible for a behavior, yet they act on different objects: one forward pass versus the parameters behind every forward pass. We ask when they agree.
We study an idealized model where a conditional computation is carried additively through a residual stream, $F(x)=F_0(x)+\sum_iα_i(x)v_i$, read out by a linear functional, and prove three exact results. First, deleting a subset of carriers collapses a matched input pair onto the same unconditional output \emph{if and only if} the removal is symmetric on the pair and leaves no outside contrast; the error is deterministic, and we give its exact form even when the two conditions hold only approximately. Second, patching a carrier moves the readout by its donor-receiver \emph{contrast}, while ablating it moves the readout by its \emph{absolute level}; neither bounds the other, and we construct pairs where every single-carrier patch flips the decision while no single-carrier ablation does. Third, for an attention head composed with its own layer's normalization and MLP, we derive an exact first-order interaction formula with a provably second-order remainder, vanishing identically when only the MLP is ablated but not, in general, when a head is.
Small transformers trained on a synthetic conditional task illustrate all three predictions: across thirty-nine ablation configurations the measured interaction is strongly rank-correlated with the idealized model's predictive accuracy (Spearman $-0.83$), and a second task and architecture reproduces the same pattern, including a further polarity reversal.
The single-block interaction result extends past one residual block, and the synthetic validation is tested against a real pretrained model, in a companion paper that takes this theory further along both axes.