Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

2026-06-22

AI Summary

This work addresses the challenge in mechanistic interpretability that, despite progress in circuit localization, component-level functional explanations remain manual and lack standardization. To this end, we propose HyVE, a novel framework that introduces language model agents into circuit explanation tasks. HyVE iteratively performs observation, hypothesis generation, and causal verification to automatically produce both component-level interpretations and circuit-level task descriptions. We construct AgenticInterpBench, the first benchmark tailored for agent-based interpretability, and evaluate HyVE across four mainstream language model architectures, demonstrating its ability to generate high-quality explanations. Our experiments reveal that causal verification constitutes the primary performance bottleneck, and we further showcase HyVE’s practical utility through a case study on arithmetic circuits in Llama-3-8B.

Abstract

Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can assist with this explanation problem once a circuit has already been identified. We introduce AgenticInterpBench, a benchmark for circuit explanation built from 84 semi-synthetic transformer circuits with 163 component-level annotations. We propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that analyzes each component through an iterative loop of observation, hypothesis generation, and causal validation, eventually producing a component-level explanation and a circuit-level task description. Across four LM backbones, HyVE recovers useful component- and task-level explanations, but no backbone is uniformly best. Our analysis shows that strong backbones usually form observation-grounded hypotheses, while failures more often arise later in the validation loop, through incomplete validation plans, code execution errors, or unresolved hypotheses. A case study on an arithmetic circuit in Llama-3-8B shows that the same formulation can extend beyond semi-synthetic benchmarks to naturally trained models. Overall, LM agents are promising circuit explainers, but reliable validation remains the key obstacle.

Problem

• mechanistic interpretability
• circuit explanation
• language model agents
• component-level explanation
• standardization

Innovation

• mechanistic interpretability
• language model agents
• circuit explanation
• HyVE
• AgenticInterpBench