MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

2026-04-16

AI Summary

This work addresses critical challenges in multi-label text classification within high-stakes medical contexts—namely, severe label imbalance, strong inter-label dependencies, complex label combinations, and the lack of robust benchmarks that are immune to data contamination and support reliable uncertainty quantification. To this end, we introduce MADE, the first dynamic multi-label classification benchmark specifically designed for medical device adverse events. Built upon temporally strict, continuously updated data splits, MADE enables joint evaluation of multi-label classification performance and uncertainty estimation. Through systematic evaluation of over twenty encoder–decoder architectures combined with diverse uncertainty quantification methods, we find that small-scale discriminative fine-tuned decoders achieve the best trade-off between head/tail label performance and calibration quality, generative fine-tuning yields the most reliable uncertainty estimates, and while large inference models enhance rare label detection, they exhibit comparatively weak uncertainty quantification capabilities.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE, a living MLTC benchmark derived from {m}edical device {ad}verse {e}vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark.

Problem

• multi-label text classification
• uncertainty quantification
• medical device adverse events
• benchmark contamination
• label imbalance

Innovation

• multi-label text classification
• uncertainty quantification
• living benchmark
• temporal data split
• medical adverse events