Foundation Models as Class-Incremental Learners for Dermatological Image Classification

2025-07-18

AI Summary

This work addresses catastrophic forgetting in class-incremental learning (CIL) for dermatological disease classification. We investigate the efficacy of large-scale pretrained foundation models in continually learning novel lesion categories without forgetting previously seen ones. We propose an efficient framework that freezes the backbone and incrementally trains only a lightweight MLP head—demonstrating, for the first time, its superior performance on dermatological image CIL. Furthermore, we introduce a parameter-free nearest prototype classifier (NPC) enabling zero-shot, zero-training inference. Our approach eliminates the need for replay buffers, regularization, or architectural expansion. Evaluated across multiple benchmark skin datasets, it achieves state-of-the-art CIL accuracy, significantly outperforming mainstream CIL paradigms. All code and data are publicly released.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Class-Incremental Learning (CIL) aims to learn new classes over time without forgetting previously acquired knowledge. The emergence of foundation models (FM) pretrained on large datasets presents new opportunities for CIL by offering rich, transferable representations. However, their potential for enabling incremental learning in dermatology remains largely unexplored. In this paper, we systematically evaluate frozen FMs pretrained on large-scale skin lesion datasets for CIL in dermatological disease classification. We propose a simple yet effective approach where the backbone remains frozen, and a lightweight MLP is trained incrementally for each task. This setup achieves state-of-the-art performance without forgetting, outperforming regularization, replay, and architecture based methods. To further explore the capabilities of frozen FMs, we examine zero training scenarios using nearest mean classifiers with prototypes derived from their embeddings. Through extensive ablation studies, we demonstrate that this prototype based variant can also achieve competitive results. Our findings highlight the strength of frozen FMs for continual learning in dermatology and support their broader adoption in real world medical applications. Our code and datasets are available here.

Problem

• Exploring foundation models for class-incremental dermatological image classification
• Evaluating frozen foundation models with lightweight MLP for continual learning
• Assessing prototype-based zero-training using foundation model embeddings

Innovation

• Frozen foundation models for incremental learning
• Lightweight MLP trained incrementally per task
• Prototype-based zero training with embeddings