Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

2026-03-18

AI Summary

This study addresses the limited evaluation of visual-language models (VLMs) on diagnosing rare skin diseases and their neglect of clinical reasoning processes. To bridge this gap, the authors introduce DermCase—the first long-context multimodal benchmark focused on rare dermatological conditions—comprising 26,030 image-text pairs and 6,354 challenging cases annotated with complete clinical information and step-by-step diagnostic reasoning chains. They further propose a novel evaluation metric based on DermLIP semantic similarity, which demonstrates high alignment with dermatologists’ judgments. Experiments across 22 state-of-the-art VLMs reveal significant deficiencies in diagnostic accuracy, differential diagnosis, and clinical reasoning. While instruction fine-tuning substantially improves performance, direct preference optimization (DPO) yields only marginal gains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.

Problem

• rare skin diseases
• diagnostic reasoning
• clinical reasoning
• vision-language models
• dermatology

Innovation

• rare skin diseases
• diagnostic reasoning
• vision-language models
• clinical benchmark
• DermCase