Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

2026-03-10

AI Summary

This work addresses the challenge of action prediction during testing when ground-truth annotations from the target viewpoint are unavailable, by proposing the first test-time adaptation framework tailored for Ego-Exo perspective alignment. The approach effectively integrates cross-view information through a multi-label prototype growth module (ML-PGM) and a vision-text dual-consistency constraint, mitigating issues arising from multiple action candidates and spatiotemporal discrepancies between viewpoints. To further enhance adaptability, the method incorporates an entropy-prioritized memory bank update strategy and a lightweight text narrative generator. Extensive experiments on the EgoMe-anti and EgoExoLearn benchmarks demonstrate significant performance gains over existing methods, validating the effectiveness and novelty of the proposed framework under the challenging setting of no target-view training data.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.

Problem

• Test-Time Adaptation
• Ego-Exo Adaptation
• Action Anticipation
• Multi-Label Learning
• Cross-View Consistency

Innovation

• Test-Time Adaptation
• Ego-Exo Adaptation
• Action Anticipation
• Multi-Label Prototype Growing
• Dual-Clue Consistency