Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification

2026-06-30

AI Summary

Test-time adaptation (TTA) under unsupervised settings often suffers from ill-posedness, leading models to converge to erroneous decision boundaries and resulting in poor stability and robustness. This work reframes TTA as a multi-hypothesis inference problem and introduces a plug-and-play particle-based framework that simultaneously explores multiple plausible parameter update trajectories. By inducing diversity across output, parameter, optimizer, and input spaces, the approach integrates entropy-minimization-based pseudo-likelihood modeling with a particle diversity mechanism to effectively mitigate ill-posedness and prevent mode collapse, while remaining compatible with existing TTA methods. Experiments demonstrate consistent performance gains of 3–4% under mixed distribution shifts, 2–3% in single-sample batch scenarios, and 1–2.5% under label shift, significantly outperforming current state-of-the-art approaches.

Abstract

Test-Time Adaptation (TTA) seeks to improve model robustness under distribution shifts by adapting parameters using unlabeled target data. However, in the absence of supervision, entropy-based adaptation is fundamentally underconstrained: multiple distinct parameter updates can achieve similarly low entropy while inducing drastically different decision boundaries. This phenomenon, known as underspecification, renders standard TTA brittle and prone to collapse into spurious modes. In this work, we reinterpret TTA through a posterior-inspired lens induced by entropy minimization, where low-entropy solutions define a pseudo-likelihood over parameters. Instead of committing to a single point estimate, we introduce a particle-based diversification framework that explores multiple plausible adaptation trajectories simultaneously. Our method can be viewed as a structured exploration of multiple plausible adaptation solutions, implemented through multi-level diversification at the output, parameter, optimizer, and input levels. Crucially, the framework acts as a plug-and-play wrapper compatible with existing TTA methods. Extensive experiments on challenging benchmarks demonstrate consistent gains in stability and robustness, achieving improvements of 3-4% under mixed shifts, 2-3% with batch size one, and 1-2.5% under label shifts, outperforming state-of-the-art baselines. Our results suggest that treating TTA as a multi-hypothesis inference problem, rather than a single-point optimization task, is key to mitigating underspecification and enabling reliable real-world deployment.

Problem

• Test-Time Adaptation
• Underspecification
• Entropy Minimization
• Distribution Shift
• Model Robustness

Innovation

• Test-Time Adaptation
• Underspecification
• Multi-Hypothesis Inference
• Entropy Minimization
• Particle-Based Diversification