Rethinking the Test-Time Prompt Tuning Objective from the Perspective of Calibration

2026-08-31

AI Summary

该研究针对测试时提示调优导致的过自信预测问题，提出了一种结合交叉熵和目标分布熵的新方法，以改善模型校准。

Abstract

Test-time prompt tuning (TPT) has emerged as a powerful paradigm, refining prompts for each test sample via entropy minimization (EM) over multiple augmented views. However, we identify a limitation in the standard EM-based adaptation: it inherently drives the model toward overconfident predictions disregarding sample-specific uncertainty, leading to significant calibration degradation. To address these limitations, we propose a new objective that replaces the conventional EM loss by aligning the original-view prediction with a target distribution derived from augmented views via cross-entropy, while adversarially incorporating the entropy of the target distribution to capture sample-specific uncertainty. Furthermore, to better construct this target distribution, we apply confidence-aware temperature scaling to each augmented-view prediction according to its confidence, sharpening confident predictions while softening uncertain ones. This formulation allows the model to increase confidence only when the target distribution is reliable, while preserving uncertainty when it reflects ambiguous or conflicting augmented-view predictions. Extensive experiments across diverse benchmarks demonstrate that our approach not only achieves state-of-the-art accuracy but also significantly improves model calibration.

Problem

• test-time prompt tuning
• entropy minimization
• calibration
• uncertainty
• overconfident predictions

Innovation

• test-time prompt tuning
• cross-entropy alignment
• sample-specific uncertainty
• confidence-aware temperature scaling