Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

2026-04-16

AI Summary

This work addresses the challenge of performing efficient and stable unsupervised test-time adaptation on black-box models accessible only via APIs. To this end, the authors propose BETA, a novel framework that, for the first time, enables query-free test-time adaptation in the black-box setting. BETA constructs a lightweight, locally white-box surrogate model to establish a differentiable path, integrating prediction alignment, consistency regularization, and prompt-learning-guided filtering—all without requiring additional API queries. Experiments demonstrate that BETA improves the accuracy of ViT-B/16 and CLIP by 7.1% and 3.4%, respectively, on ImageNet-C. Moreover, when applied to commercial APIs, BETA achieves performance comparable to ZOO with 250× fewer queries, offering a compelling balance between efficiency and real-time applicability.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but faces high query costs and optimization challenges in the unsupervised TTA setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that addresses these limitations by employing a lightweight, local white-box steering model to create a tractable gradient pathway. Through a prediction harmonization technique combined with consistency regularization and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency beyond standard inference. On ImageNet-C, BETA achieves a +7.1% accuracy gain on ViT-B/16 and +3.4% on CLIP, surpassing strong white-box and gray-box methods including TENT and TPT. On a commercial API, BETA achieves comparable performance to ZOO at 250x lower cost while maintaining real-time inference speed, establishing it as a practical and efficient solution for real-world black-box TTA.

Problem

• Test-Time Adaptation
• Black-Box Models
• Zeroth-Order Optimization
• API-based Inference
• Unsupervised Adaptation

Innovation

• Test-Time Adaptation
• Black-Box Models
• Zeroth-Order Optimization
• Prompt Learning
• Consistency Regularization