A multimodal slice discovery framework for systematic failure detection and explanation in medical image classification

2026-02-27

AI Summary

This work proposes the first multimodal slice discovery framework tailored for medical applications, addressing the limitations of existing auditing methods that rely primarily on unimodal features or metadata subgroup analyses and thus fail to comprehensively identify and interpret systematic model failures. By extending slice discovery into multimodal representation spaces, the framework integrates heterogeneous sources—such as medical images and associated clinical text—to automatically detect and explain systematic failure modes in image classifiers. Combining multimodal representation learning with interpretable slice discovery algorithms, the approach significantly outperforms unimodal baselines on the MIMIC-CXR-JPG dataset, enabling more comprehensive, accurate, and interpretable model auditing.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Despite advances in machine learning-based medical image classifiers, the safety and reliability of these systems remain major concerns in practical settings. Existing auditing approaches mainly rely on unimodal features or metadata-based subgroup analyses, which are limited in interpretability and often fail to capture hidden systematic failures. To address these limitations, we introduce the first automated auditing framework that extends slice discovery methods to multimodal representations specifically for medical applications. Comprehensive experiments were conducted under common failure scenarios using the MIMIC-CXR-JPG dataset, demonstrating the framework's strong capability in both failure discovery and explanation generation. Our results also show that multimodal information generally allows more comprehensive and effective auditing of classifiers, while unimodal variants beyond image-only inputs exhibit strong potential in scenarios where resources are constrained.

Problem

• systematic failure detection
• medical image classification
• multimodal slice discovery
• auditing
• failure explanation

Innovation

• multimodal slice discovery
• systematic failure detection
• medical image classification
• automated auditing
• failure explanation