RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons

2026-08-25

AI Summary

为解决神经元在大范围内的稳定行为发现难题，提出RACE方法，通过前向传递统计框架评估Transformer神经元的功能一致性，具有更好的领域特异性和计算效率。

Abstract

Discovering stable neuron behavior across entire domains remains a challenge in mechanistic interpretability. Existing methods often rely on instance-level point estimates or computationally expensive procedures, which either obscure population-level variability or limit scalable domain-wide analysis. We present RACE (Residual Alignment for Consistency Estimation), a forward-pass statistical framework that evaluates the domain-wide functional consistency of Transformer neurons. Perturbation experiments demonstrate that RACE achieves superior domain specificity compared to gradient-based point estimates. Meanwhile, token-distribution-level results verify the association between the selected neurons and the target domain. Furthermore, its computational overhead is two orders of magnitude lower than that of gradient-based methods.

Problem

• mechanistic interpretability
• neuron behavior
• domain-wide analysis
• computational overhead

Innovation

• RACE
• functional consistency
• domain-wide analysis
• computational efficiency