DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMs

2026-01-05

arXiv.org

AI Summary

This work addresses the limitations of dermatology multimodal large language models (MLLMs)—namely, scarce training data, narrow task coverage, and insufficient alignment with clinical diagnostic workflows—by introducing DermoInstruct, a morphology-anchored instruction dataset comprising 210,000 images and 770,000 diagnostic trajectories. The authors propose DermoBench, a comprehensive evaluation benchmark, and develop DermoGPT, the first open MLLM framework in dermatology that anchors on skin lesion morphology and spans the full clinical reasoning chain. DermoGPT aligns visual observations with diagnostic conclusions through supervised fine-tuning and Morphologically-Anchored Visual-Inference-Consistent (MAVIC) reinforcement learning, while enhancing robustness via Confidence-Consistency Test-time adaptation (CCT). Evaluated on 11 tasks in DermoBench, DermoGPT significantly outperforms 16 baselines, achieving state-of-the-art performance across morphology, diagnosis, reasoning, and fairness dimensions, thereby substantially narrowing the human–AI performance gap.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal Large Language Models (MLLMs) show promise for medical applications, yet progress in dermatology lags due to limited training data, narrow task coverage, and lack of clinically-grounded supervision that mirrors expert diagnostic workflows. We present a comprehensive framework to address these gaps. First, we introduce DermoInstruct, a large-scale morphology-anchored instruction corpus comprising 211,243 images and 772,675 trajectories across five task formats, capturing the complete diagnostic pipeline from morphological observation and clinical reasoning to final diagnosis. Second, we establish DermoBench, a rigorous benchmark evaluating 11 tasks across four clinical axes: Morphology, Diagnosis, Reasoning, and Fairness, including a challenging subset of 3,600 expert-verified open-ended instances and human performance baselines. Third, we develop DermoGPT, a dermatology reasoning MLLM trained via supervised fine-tuning followed by our Morphologically-Anchored Visual-Inference-Consistent (MAVIC) reinforcement learning objective, which enforces consistency between visual observations and diagnostic conclusions. At inference, we deploy Confidence-Consistency Test-time adaptation (CCT) for robust predictions. Experiments show DermoGPT significantly outperforms 16 representative baselines across all axes, achieving state-of-the-art performance while substantially narrowing the human-AI gap. DermoInstruct, DermoBench and DermoGPT will be made publicly available at https://github.com/mendicant04/DermoGPT upon acceptance.

Problem

• Multimodal Large Language Models
• Dermatology
• Morphology-Grounded Reasoning
• Clinical Diagnosis
• Training Data Scarcity

Innovation

• Morphologically-Anchored Visual-Inference-Consistent (MAVIC)
• Confidence-Consistency Test-time adaptation (CCT)
• DermoInstruct
• DermoBench
• Dermatology MLLM