MADPOT: Medical Anomaly Detection with CLIP Adaptation and Partial Optimal Transport

2025-07-09

AI Summary

Medical anomaly detection faces challenges including diverse imaging modalities, substantial anatomical variability, and severe scarcity of annotated data. To address these, we propose a CLIP-adaptation framework that requires neither synthetic data nor external memory banks. Our method introduces vision adapters and multi-prompt learning, coupled with Partial Optimal Transport (POT) for fine-grained local feature alignment to precisely capture subtle pathological patterns; it further integrates contrastive learning to enhance intra-class compactness and inter-class separability. Crucially, our approach overcomes the limitations of single-prompt paradigms by explicitly modeling anatomical heterogeneity and cross-modal discrepancies. Extensive experiments demonstrate state-of-the-art performance under few-shot, zero-shot, and cross-dataset settings, significantly improving model robustness and generalization across diverse medical imaging domains.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Medical anomaly detection (AD) is challenging due to diverse imaging modalities, anatomical variations, and limited labeled data. We propose a novel approach combining visual adapters and prompt learning with Partial Optimal Transport (POT) and contrastive learning (CL) to improve CLIP's adaptability to medical images, particularly for AD. Unlike standard prompt learning, which often yields a single representation, our method employs multiple prompts aligned with local features via POT to capture subtle abnormalities. CL further enforces intra-class cohesion and inter-class separation. Our method achieves state-of-the-art results in few-shot, zero-shot, and cross-dataset scenarios without synthetic data or memory banks. The code is available at https://github.com/mahshid1998/MADPOT.

Problem

• Improving CLIP's adaptability to medical anomaly detection
• Addressing diverse imaging modalities and limited labeled data
• Capturing subtle abnormalities via multi-prompt POT alignment

Innovation

• Uses visual adapters and prompt learning
• Employs multiple prompts with POT alignment
• Combines contrastive learning for better separation