Challenges in interpretability of additive models

2025-04-14

AI Summary

This paper exposes a fundamental flaw in generalized additive models (GAMs), including neural additive models, when claimed to be “interpretable” and “suitable for safety-critical applications”: they suffer from triple non-identifiability—of parameters, component functions, and model structure—leading to non-unique local and global attributions and severely undermining interpretability’s reliability. Method: We systematically classify and rigorously prove how each type of non-identifiability fundamentally compromises interpretability, integrating statistical identifiability theory, functional space analysis, counterexample construction, and sensitivity analysis; we further propose an interpretability evaluation framework grounded in identifiability. Contribution/Results: We establish “identifiability as a prerequisite for interpretability” as a formal theoretical principle. Our work provides critical theoretical boundaries and practical warnings for trustworthy AI model selection, highlighting that unaddressed non-identifiability invalidates attribution-based explanations—even in ostensibly transparent models.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We review generalized additive models as a type of ``transparent'' model that has recently seen renewed interest in the deep learning community as neural additive models. We highlight multiple types of nonidentifiability in this model class and discuss challenges in interpretability, arguing for restraint when claiming ``interpretability'' or ``suitability for safety-critical applications'' of such models.

Problem

• Addressing interpretability challenges in additive models
• Identifying nonidentifiability issues in neural additive models
• Evaluating suitability for safety-critical applications

Innovation

• Generalized additive models for transparency
• Neural additive models in deep learning
• Addressing nonidentifiability in model interpretability