Do Natural Language Descriptions of Model Activations Convey Privileged Information?

2025-09-16

AI Summary

This paper challenges whether existing “verbalizer LLM”-based activation interpretation methods genuinely uncover the internal representational mechanisms of target large language models (LLMs), or merely reproduce input information or the interpreter’s parametric priors. Method: The authors design controlled experiments and cross-dataset benchmarks to systematically disentangle three confounding factors: input semantics, target-model activation states, and interpreter-model biases. Contribution/Results: Key findings show that mainstream methods pass standard evaluation benchmarks even when target-model activations are entirely ablated—indicating their outputs stem predominantly from the interpreter’s parametric knowledge rather than the target model’s private representations. Consequently, the paper identifies a critical lack of discriminative power in current evaluation datasets and introduces the first dedicated evaluation framework for attribution of explanation provenance. This work provides both methodological reflection and an empirical benchmark for interpretability research.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually provide privileged knowledge about the internal workings of the target model, or do they merely convey information about its inputs? We critically evaluate popular verbalization methods across datasets used in prior work and find that they succeed at benchmarks without any access to target model internals, suggesting that these datasets are not ideal for evaluating verbalization methods. We then run controlled experiments which reveal that verbalizations often reflect the parametric knowledge of the verbalizer LLM which generated them, rather than the activations of the target LLM being decoded. Taken together, our results indicate a need for targeted benchmarks and experimental controls to rigorously assess whether verbalization methods provide meaningful insights into the operations of LLMs.

Problem

• Evaluating if activation verbalization reveals model internals
• Assessing whether verbalizations reflect target model or verbalizer knowledge
• Identifying need for controlled benchmarks in interpretability methods

Innovation

• Evaluates verbalization methods using controlled experiments
• Reveals verbalizer LLM's parametric knowledge dominates descriptions
• Proposes targeted benchmarks for meaningful activation insights