Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

2026-08-17

AI Summary

This study addresses the lack of effective evaluation for large language model explanations and their failure to enhance counterfactual prediction by proposing the CHIVE framework. Introducing a novel perspective of counterfactual simulatability, this method employs an agent-based pipeline to automatically generate explanations and evidence while constructing counterfactual training data. Experiments involving thousands of high-quality samples demonstrate that traditional interpretability techniques yield no predictive gains, whereas counterfactual training significantly improves model generalization in out-of-distribution scenarios. Consequently, this work establishes a new paradigm for assessing explanation quality and validates the critical value of counterfactual data in enhancing model robustness.

Abstract

Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.

Problem

• LLM interpretability
• counterfactual simulatability
• explanation evaluation
• chain of thought faithfulness

Innovation

• Counterfactual Simulatability
• CHIVE
• Agentic Pipeline
• LLM Interpretability
• Out-of-Distribution Generalization