Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation

2025-08-11

AI Summary

Existing single-layer linear classifiers struggle to robustly handle complex and diverse distribution shifts during test-time adaptation. To address this, we propose the Hierarchical Adaptation Network (HAN) coupled with a task-vector coordination framework. Our key contributions are: (1) a dynamic layer selection mechanism that adaptively activates the optimal feature layer based on input complexity; (2) cross-layer weight fusion with consistency gating to suppress noise and enhance output stability of linear classifiers; and (3) task-vector fine-tuning grounded in hierarchical representation decomposition, enabling plug-and-play integration. Evaluated across multiple target-domain datasets, HAN consistently outperforms state-of-the-art methods—particularly under challenging conditions such as small batch sizes, high anomaly rates, and elevated uncertainty—demonstrating superior robustness and generalization capability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Test-time adaptation allows pretrained models to adjust to incoming data streams, addressing distribution shifts between source and target domains. However, standard methods rely on single-dimensional linear classification layers, which often fail to handle diverse and complex shifts. We propose Hierarchical Adaptive Networks with Task Vectors (Hi-Vec), which leverages multiple layers of increasing size for dynamic test-time adaptation. By decomposing the encoder's representation space into such hierarchically organized layers, Hi-Vec, in a plug-and-play manner, allows existing methods to adapt to shifts of varying complexity. Our contributions are threefold: First, we propose dynamic layer selection for automatic identification of the optimal layer for adaptation to each test batch. Second, we propose a mechanism that merges weights from the dynamic layer to other layers, ensuring all layers receive target information. Third, we propose linear layer agreement that acts as a gating function, preventing erroneous fine-tuning by adaptation on noisy batches. We rigorously evaluate the performance of Hi-Vec in challenging scenarios and on multiple target datasets, proving its strong capability to advance state-of-the-art methods. Our results show that Hi-Vec improves robustness, addresses uncertainty, and handles limited batch sizes and increased outlier rates.

Problem

• Handles diverse distribution shifts in test-time adaptation
• Dynamic layer selection for optimal adaptation
• Prevents erroneous fine-tuning on noisy data batches

Innovation

• Hierarchical adaptive networks for dynamic adaptation
• Dynamic layer selection for optimal adaptation
• Linear layer agreement prevents erroneous fine-tuning