USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

2026-07-04

AI Summary

This work addresses the performance degradation of vision-language models under distribution shifts during test time by proposing a Unified Self-Ensemble (USE) framework. USE employs an adaptive self-ensembling strategy jointly during optimization and inference, leveraging test images and their augmented views to generate more reliable pseudo-labels and enforce consistency. The approach reveals the implicit reliance of Test-Time Prompting (TPT) on pseudo-label quality, unifies the objectives of optimization and inference, and functions as a lightweight, training-free test-time adaptation (TTA) method that can be used standalone or to augment existing techniques. Experiments demonstrate that USE significantly outperforms current TTA methods across multiple benchmarks, with its self-ensembling component consistently yielding performance gains whether applied independently or in combination with other approaches.

Abstract

Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.

Problem

• Test-Time Adaptation
• Prompt Tuning
• Vision-Language Models
• Self-Ensembling
• Pseudo Labels

Innovation

• self-ensembling
• test-time adaptation
• prompt tuning
• pseudo labels
• vision-language models