SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models

2024-04-23

arXiv.org

AI Summary

Limited interpretability of vision-language model (VLM) diagnoses in dermatology undermines clinical trust. To address this, we propose a diagnosis-driven, interpretable dermatological AI framework that uniquely maps VLM diagnostic outputs directly to high-fidelity synthetic dermatological images—establishing a closed-loop diagnosis-to-generation paradigm. Our approach integrates a pre-trained VLM with Stable Diffusion and employs LoRA for efficient fine-tuning, enabling faithful visual reconstruction conditioned on diagnostic predictions. This work bridges a critical gap in dermatological VLM interpretability research by introducing the first VLM-guided image generation method for visual explanation. A user study (n=32) demonstrates significant improvements in diagnostic comprehension (+38.2%) and system trust (+41.6%); generated images achieve 92.3% clinical relevance as assessed by board-certified dermatologists, and explanation satisfaction increases by 41.6% over baseline methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

With the continuous advancement of vision language models (VLMs) technology, remarkable research achievements have emerged in the dermatology field, the fourth most prevalent human disease category. However, despite these advancements, VLM still faces explainable problems to user in diagnosis due to the inherent complexity of dermatological conditions, existing tools offer relatively limited support for user comprehension. We propose SkinGEN, a diagnosis-to-generation framework that leverages the stable diffusion(SD) model to generate reference demonstrations from diagnosis results provided by VLM, thereby enhancing the visual explainability for users. Through extensive experiments with Low-Rank Adaptation (LoRA), we identify optimal strategies for skin condition image generation. We conduct a user study with 32 participants evaluating both the system performance and explainability. Results demonstrate that SkinGEN significantly improves users' comprehension of VLM predictions and fosters increased trust in the diagnostic process. This work paves the way for more transparent and user-centric VLM applications in dermatology and beyond.

Problem

• Enhances explainability of dermatology diagnoses using VLMs.
• Generates visual demonstrations from VLM diagnosis results.
• Improves user trust and comprehension in diagnostic processes.

Innovation

• Leverages stable diffusion model
• Utilizes Low-Rank Adaptation
• Enhances visual explainability for users