Exploring Weak-to-Strong Generalization for CLIP-based Classification

2025-11-23

AI Summary

This work addresses the heavy reliance of CLIP-style models on extensive human annotations for fine-grained or complex categories. To reduce dependence on costly manual supervision, we propose a weak-model-supervised strong-model classification paradigm. Our core method, Class Prototype Learning (CPL), leverages pseudo-labels generated by a lightweight weak model to construct robust and transferable class prototypes, which are then refined via contrastive learning to enhance vision-language alignment. Crucially, CPL is the first approach to successfully extend weak-to-strong generalization to multimodal vision-language settings. Extensive experiments under constrained regimes—including few-shot classification and low-resource pretraining—demonstrate that CPL consistently outperforms strong baselines, achieving an average accuracy gain of 3.67% across benchmarks. The improvement is particularly pronounced under annotation scarcity, validating CPL’s efficacy in low-supervision scenarios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient. A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67% improvement over strong baseline methods.

Problem

• Extending weak-to-strong generalization to vision-language CLIP models
• Enhancing CLIP classification with limited pretraining data
• Learning representative class prototypes under weak supervision

Innovation

• Uses weak model to supervise stronger model
• Introduces class prototype learning method
• Enhances CLIP classification with limited pretraining