Mechanistic Interpretability as Statistical Estimation: A Variance Analysis of EAP-IG

2025-10-01

AI Summary

This study addresses the core limitations of circuit-discovery methods in mechanistic interpretability (MI)—notably high structural variance, hyperparameter sensitivity, and unreliable outputs—exemplified by EAP-IG. We first formalize interpretability methods as statistical estimators and conduct a systematic, multi-dimensional stability analysis across five perturbation axes: input resampling, prompt rewriting, causal ablation, noise injection, and hyperparameter sweeps. Experiments reveal pervasive structural instability and poor robustness of EAP-IG across diverse models and tasks. Based on these findings, we propose stability as a fundamental scientific criterion for evaluating mechanistic interpretability, advocate for standardized stability evaluation protocols, and recommend their integration into foundational MI research practices. This work establishes both a theoretical framework and empirical benchmarks to enhance the credibility, reproducibility, and scientific rigor of MI methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The development of trustworthy artificial intelligence requires moving beyond black-box performance metrics toward an understanding of models' internal computations. Mechanistic Interpretability (MI) aims to meet this need by identifying the algorithmic mechanisms underlying model behaviors. Yet, the scientific rigor of MI critically depends on the reliability of its findings. In this work, we argue that interpretability methods, such as circuit discovery, should be viewed as statistical estimators, subject to questions of variance and robustness. To illustrate this statistical framing, we present a systematic stability analysis of a state-of-the-art circuit discovery method: EAP-IG. We evaluate its variance and robustness through a comprehensive suite of controlled perturbations, including input resampling, prompt paraphrasing, hyperparameter variation, and injected noise within the causal analysis itself. Across a diverse set of models and tasks, our results demonstrate that EAP-IG exhibits high structural variance and sensitivity to hyperparameters, questioning the stability of its findings. Based on these results, we offer a set of best-practice recommendations for the field, advocating for the routine reporting of stability metrics to promote a more rigorous and statistically grounded science of interpretability.

Problem

• Evaluating reliability of mechanistic interpretability methods as statistical estimators
• Analyzing variance and robustness in neural network circuit discovery techniques
• Assessing stability of EAP-IG method through systematic perturbation experiments

Innovation

• Viewing interpretability methods as statistical estimators
• Conducting systematic stability analysis of EAP-IG method
• Evaluating variance through controlled perturbations and noise