Automated Interpretability and Feature Discovery in Language Models with Agents

2026-05-02

AI Summary

This work addresses the limited interpretability of large language models, which hinders the automatic discovery and validation of their internal functional features. To this end, we propose the first end-to-end automated, multi-agent interpretability framework that generates auditable and falsifiable mechanistic explanations through coupled “explanation optimization” and “feature discovery” loops. Our approach integrates activation-space k-nearest neighbor graphs, statistical separability analysis, semantic consistency evaluation, and targeted prompt control. Evaluated on MLP neurons in Gemma-2 variants and sparse-weight Transformers, the method substantially outperforms existing one-shot automated explanation techniques, successfully identifying key internal features related to language specificity and safety while producing traceable explanatory pathways.

Abstract

We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models. The system runs two coupled loops: (1) explanation refinement, where an agent proposes competing hypotheses and iteratively tests them with targeted prompt controls and a multi-metric evaluation; and (2) feature discovery, where an agent generates prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP neurons in weight-sparse transformers, our agent improves over one-shot auto-interpretations, discovers language-specific and safety-relevant features, and produces auditable explanation traces, showing that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.

Problem

• mechanistic interpretability
• feature discovery
• large language models
• automated explanation
• activation space

Innovation

• autonomous multiagent framework
• mechanistic interpretability
• feature discovery
• explanation refinement
• activation space