The Anatomy of Evidence: An Investigation Into Explainable ICD Coding

2025-07-02

AI Summary

Existing automatic ICD coding systems lack interpretability for clinical long-text, multi-label scenarios, and current evaluation methods fail to validate evidence–code consistency. Method: Leveraging the MDACE dataset, we propose a novel evidence matching metric that quantifies semantic overlap between model-extracted textual evidence and ground-truth code descriptions, integrating text matching and evidence alignment techniques to systematically assess the effectiveness and bias of mainstream interpretability methods in evidence extraction. Contribution/Results: Our evaluation identifies successful and failure cases, revealing that while current methods capture partial ground-truth evidence, overall evidence–code consistency remains limited. The proposed metric significantly enhances objectivity and clinical plausibility in interpretability assessment. This work establishes a reproducible, clinically grounded evaluation framework—supporting the development, diagnostic traceability, and trustworthy deployment of interpretable medical coding systems.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Automatic medical coding has the potential to ease documentation and billing processes. For this task, transparency plays an important role for medical coders and regulatory bodies, which can be achieved using explainability methods. However, the evaluation of these approaches has been mostly limited to short text and binary settings due to a scarcity of annotated data. Recent efforts by Cheng et al. (2023) have introduced the MDACE dataset, which provides a valuable resource containing code evidence in clinical records. In this work, we conduct an in-depth analysis of the MDACE dataset and perform plausibility evaluation of current explainable medical coding systems from an applied perspective. With this, we contribute to a deeper understanding of automatic medical coding and evidence extraction. Our findings reveal that ground truth evidence aligns with code descriptions to a certain degree. An investigation into state-of-the-art approaches shows a high overlap with ground truth evidence. We propose match measures and highlight success and failure cases. Based on our findings, we provide recommendations for developing and evaluating explainable medical coding systems.

Problem

• Evaluating explainable ICD coding systems using MDACE dataset
• Assessing alignment between ground truth evidence and code descriptions
• Proposing measures for improving explainable medical coding systems

Innovation

• Utilizes explainability methods for medical coding
• Analyzes MDACE dataset for evidence extraction
• Proposes match measures for evaluation