Annotation-Efficient Active Test-Time Adaptation with Conformal Prediction

2025-09-29

AI Summary

To address the inefficiency in sample selection and high annotation cost caused by heuristic uncertainty measures in Adaptive Test-Time Adaptation (ATTA) under domain shift, this paper introduces SmoothCP—the first integration of coverage-guaranteed conformal prediction into the ATTA framework. SmoothCP comprises four key components: (i) smoothed conformal scores for robust uncertainty quantification, (ii) pseudo-coverage-driven online weight updating, (iii) lightweight domain shift detection, and (iv) a phased hybrid update mechanism. By leveraging theoretically grounded uncertainty calibration, SmoothCP significantly improves critical sample identification accuracy and annotation efficiency. Extensive experiments demonstrate that SmoothCP consistently outperforms state-of-the-art methods by approximately 5% in accuracy across multiple benchmarks. Moreover, under identical annotation budgets, it achieves superior performance, effectively reducing human annotation effort while maintaining rigorous coverage guarantees.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

Active Test-Time Adaptation (ATTA) improves model robustness under domain shift by selectively querying human annotations at deployment, but existing methods use heuristic uncertainty measures and suffer from low data selection efficiency, wasting human annotation budget. We propose Conformal Prediction Active TTA (CPATTA), which first brings principled, coverage-guaranteed uncertainty into ATTA. CPATTA employs smoothed conformal scores with a top-K certainty measure, an online weight-update algorithm driven by pseudo coverage, a domain-shift detector that adapts human supervision, and a staged update scheme balances human-labeled and model-labeled data. Extensive experiments demonstrate that CPATTA consistently outperforms the state-of-the-art ATTA methods by around 5% in accuracy. Our code and datasets are available at https://github.com/tingyushi/CPATTA.

Problem

• Improves model robustness under domain shift
• Enhances data selection efficiency for annotations
• Provides principled uncertainty measures with coverage guarantees

Innovation

• Uses conformal scores for principled uncertainty measurement
• Implements online weight updates with pseudo coverage
• Employs staged updates balancing labeled and unlabeled data