EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample

2026-05-15

AI Summary

This work addresses the challenge of test-time model adaptation under stringent deployment constraints, where existing methods relying on backpropagation are infeasible for edge devices, quantized models, or black-box settings. The authors propose an efficient, backpropagation-free adaptation method that operates under the strict requirement of only two forward passes per sample. By leveraging loss scale invariance to avoid shortcut solutions, anchor-guided optimization to mitigate weight drift, and sample-wise symmetric bidirectional perturbations to estimate update directions, the approach builds upon a zeroth-order optimization framework. Evaluated on ImageNet-C with a ViT-Base backbone, the method significantly outperforms both backpropagation-based DeYO and backpropagation-free FOA, while achieving a 14× speedup in inference time over FOA.

Abstract

Test-time model evolution offers a promising way for deployed models to improve from unlabeled test-time experience, yet most existing methods depend on backpropagation (BP), which incurs substantial memory overhead and makes them difficult to deploy on edge devices, quantized models, specialized accelerators, or black-box models. In this work, we study test-time model evolution under a strict two-forward budget, a setting that pushes adaptation toward highly efficient real-world deployment. We reveal three key obstacles in zeroth-order test-time optimization: susceptibility to shortcut solutions, uncontrolled weight drift, and ineffective update direction estimation. To overcome them, we propose EVA-0, a minimal zeroth-order adaptation framework that: 1) keeps the loss scale-invariant to prevent shortcut solutions; 2) devises an anchor-guided optimization strategy to alleviate weight drift; 3) uses sample-wise symmetric two-sided perturbation for update direction estimation and inference. EVA-0 requires no BP and performs both inference and adaptation within only two forward passes per sample. Results on ImageNet-C & ViT-Base show that EVA-0 outperforms both BP-based DeYO and BP-free FOA, while achieving a 14x speed-up over FOA. Code will be released.

Problem

• test-time model evolution
• zeroth-order optimization
• forward-only adaptation
• model deployment
• weight drift

Innovation

• test-time adaptation
• zeroth-order optimization
• forward-only inference
• model evolution
• edge deployment