The Query Channel: Information-Theoretic Limits of Masking-Based Explanations

2026-04-17

AI Summary

This work reveals fundamental limitations of mask-based explanation methods—such as LIME and KernelSHAP—in reliably recovering feature importance under a limited query budget. It formulates black-box explanation as a communication problem, modeling the true explanation as a message to be transmitted through a query channel, where each masked evaluation constitutes a channel use. Introducing the notions of “explanation rate” and “identification capacity,” the paper establishes strong converse and achievability results that characterize the theoretical limits of reliable explanation. Through information-theoretic analysis, entropy-based modeling of hypothesis classes, and empirical validation, it demonstrates that reliable recovery is achievable when the explanation rate lies below channel capacity, whereas standard convex surrogate approaches still fail under moderate query budgets. The study further uncovers performance cliffs and error floors induced by noise and nonlinearity, which fundamentally constrain high-resolution interpretability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Interpretability, Explainability, and Transparency
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Masking-based post-hoc explanation methods, such as KernelSHAP and LIME, estimate local feature importance by querying a black-box model under randomized perturbations. This paper formulates this procedure as communication over a query channel, where the latent explanation acts as a message and each masked evaluation is a channel use. Within this framework, the complexity of the explanation is captured by the entropy of the hypothesis class, while the query interface supplies information at a rate determined by an identification capacity per query. We derive a strong converse showing that, if the explanation rate exceeds this capacity, the probability of exact recovery necessarily converges to one in error for any sequence of explainers and decoders. We also prove an achievability result establishing that a sparse maximum-likelihood decoder attains reliable recovery when the rate lies below capacity. A Monte Carlo estimator of mutual information yields a non-asymptotic query benchmark that we use to compare optimal decoding with Lasso- and OLS-based procedures that mirror LIME and KernelSHAP. Experiments reveal a range of query budgets where information theory permits reliable explanations but standard convex surrogates still fail. Finally, we interpret super-pixel resolution and tokenization for neural language models as a source-coding choice that sets the entropy of the explanation and show how Gaussian noise and nonlinear curvature degrade the query channel, induce waterfall and error-floor behavior, and render high-resolution explanations unattainable.

Problem

• masking-based explanations
• query channel
• information-theoretic limits
• feature importance
• black-box model

Innovation

• query channel
• information-theoretic limits
• masking-based explanations
• identification capacity
• sparse maximum-likelihood decoding