Sparse Inter-Layer Dependencies of Transformer FFN Neurons

2026-07-13

AI Summary

This work addresses the challenge of interpreting dependencies among neurons in Transformer feedforward networks (FFNs), which are obscured by residual connections that superimpose activation signals. The authors propose a training-free attribution method that combines activation masking with propagation analysis to efficiently identify sparse, structured dependencies of FFN neurons on upstream attention outputs and preceding-layer neurons. Experiments demonstrate that only a small subset of critical upstream activations is sufficient to faithfully reconstruct target neuron responses. Moreover, applying the proposed mask network-wide at moderate sparsity levels incurs negligible impact on model perplexity. This study presents the first neuron-level characterization of cross-layer sparse dependency structures within FFNs and introduces a scalable, circuit-level interpretability tool for deep Transformer models.

Abstract

Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream. We examine whether the activation of an FFN neuron can be explained by a sparse set of preceding neuron activations and attention outputs. We introduce a training-free attribution method that estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation. Empirically, across models and layers, we find that small subsets of preceding activations and attention outputs suffice to preserve neuron activations with high fidelity when all remaining inputs are masked with their average values. Effective sparsity is even greater when accounting for the inherent activation sparsity of upstream layers. Moreover, applying the neuron-specific masks in all layers simultaneously, such that the induced deviations propagate through the network, leaves model perplexity largely unchanged at moderate sparsity levels. These results demonstrate that, despite dense parameterization, FFNs exhibit sparse and structured inter-layer dependencies at the neuron level. Our method provides a practical, scalable tool for circuit-level interpretability and identifies candidate sparse pathways with potential implications for efficient inference.

Problem

• Transformer
• Feedforward Network
• Neuron Activation
• Inter-layer Dependencies
• Sparsity

Innovation

• sparse inter-layer dependencies
• feedforward network interpretability
• training-free attribution
• neuron activation sparsity
• efficient inference