NLPrompt: Noise-Label Prompt Learning for Vision-Language Models

2024-12-02

arXiv.org

AI Summary

To address the performance degradation of vision-language foundation models (e.g., CLIP) under noisy label supervision in prompt learning, this paper proposes a robust prompt learning framework for noisy labels. Methodologically, it integrates text-encoder prototype representations, prompt tuning, and noise-aware loss design. Its key contributions are: (1) PromptMAE—introducing Mean Absolute Error (MAE) loss into prompt learning for the first time to mitigate gradient distortion caused by label noise; and (2) PromptOT—a prompt-based data purification method grounded in optimal transport theory, enabling noise-aware sample reweighting and prototype-level label correction. Extensive experiments demonstrate that the framework significantly improves classification accuracy and robustness across diverse noise rates and types, while enhancing signal-to-noise ratio and cross-domain generalization capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The emergence of vision-language foundation models, such as CLIP, has revolutionized image-text representation, enabling a broad range of applications via prompt learning. Despite its promise, real-world datasets often contain noisy labels that can degrade prompt learning performance. In this paper, we demonstrate that using mean absolute error (MAE) loss in prompt learning, named PromptMAE, significantly enhances robustness against noisy labels while maintaining high accuracy. Though MAE is straightforward and recognized for its robustness, it is rarely used in noisy-label learning due to its slow convergence and poor performance outside prompt learning scenarios. To elucidate the robustness of PromptMAE, we leverage feature learning theory to show that MAE can suppress the influence of noisy samples, thereby improving the signal-to-noise ratio and enhancing overall robustness. Additionally, we introduce PromptOT, a prompt-based optimal transport data purification method to enhance the robustness further. PromptOT employs text encoder representations in vision-language models as prototypes to construct an optimal transportation matrix. This matrix effectively partitions datasets into clean and noisy subsets, allowing for the application of cross-entropy loss to the clean subset and MAE loss to the noisy subset. Our Noise-Label Prompt Learning method, named NLPrompt, offers a simple and efficient approach that leverages the expressive representation and precise alignment capabilities of vision-language models for robust prompt learning. We validate NLPrompt through extensive experiments across various noise settings, demonstrating significant performance improvements.

Problem

• Enhances robustness against noisy labels in prompt learning
• Introduces PromptMAE for noise suppression in vision-language models
• Proposes PromptOT for data purification in noisy-label learning

Innovation

• Uses MAE loss for robust noisy-label prompt learning
• Introduces PromptOT for optimal transport data purification
• Combines cross-entropy and MAE losses for subsets