🤖 AI Summary
This work addresses the lack of reliable uncertainty estimation in CLIP under few-shot and out-of-distribution (OOD) settings by proposing GP-Adapter, a training-free, plug-and-play framework. GP-Adapter constructs modality-specific one-class Gaussian processes on frozen CLIP features—employing an RBF kernel for images and a linear kernel for text—and fuses predictive statistics from both modalities to produce variance-aware confidence scores. As the first approach to integrate Gaussian process-based uncertainty modeling into CLIP, GP-Adapter enables effective few-shot classification and OOD detection without fine-tuning, incurs low memory overhead, and complements existing prompt-learning methods. Experiments demonstrate competitive classification performance on ImageNet and multiple OOD benchmarks, along with significantly improved OOD detection accuracy.
📝 Abstract
We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection. While CLIP achieves strong zero-shot recognition, it yields deterministic similarity scores and offers limited uncertainty information, which is critical under distribution shift and data scarcity. GP-Adapter constructs modality-specific, class-wise one-class GPs on top of frozen CLIP embeddings using an RBF kernel for image features and a linear kernel for text prompts and fuses their predictive statistics to produce a variance-aware confidence score for OOD detection. The method requires no fine-tuning of the CLIP backbone and relies only on a small $K$-shot cache and lightweight hyperparameter selection, with memory cost scaling as $O(CK^2)$ for $C$ classes and $K$ shots. Experiments on ImageNet and multiple OOD benchmarks show that GP-Adapter provides competitive few-shot performance and consistently improves OOD detection when combined with prompt-learning baselines, highlighting the complementarity between GP-based uncertainty modeling and prompt learning. Overall, our results suggest that integrating probabilistic inference with large pre-trained vision-language models can improve reliability in low-data and distribution-shifted settings. Code is available at https://github.com/tms-byte/GP-Adapter