Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception

2025-12-29

AI Summary

Existing end-to-end 3D perception methods rely on a uniform physical motion model—e.g., constant velocity—combined with semantic features for spatiotemporal alignment, failing to accommodate the heterogeneity of object categories and dynamic motions, thus introducing alignment bias. To address this, we propose an adaptive multi-hypothesis spatiotemporal alignment mechanism and introduce HAT (Hypothesis-Aware Transformer), the first unsupervised module enabling object-level dynamic selection of optimal motion models by jointly leveraging semantic and motion cues from cached queries. Our approach integrates multiple explicit motion models, motion-aware feature proposal generation, and semantic-motion joint embedding decoding, yielding a fully differentiable, end-to-end learnable alignment framework. On nuScenes, our method achieves an AMOTA of 46.0%, outperforming the baseline by +3.1%, improves mAP by +1.3%, reduces collision rate by 32%, and demonstrates significantly enhanced robustness under semantic-degraded conditions.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Spatio-temporal alignment is crucial for temporal modeling of end-to-end (E2E) perception in autonomous driving (AD), providing valuable structural and textural prior information. Existing methods typically rely on the attention mechanism to align objects across frames, simplifying the motion model with a unified explicit physical model (constant velocity, etc.). These approaches prefer semantic features for implicit alignment, challenging the importance of explicit motion modeling in the traditional perception paradigm. However, variations in motion states and object features across categories and frames render this alignment suboptimal. To address this, we propose HAT, a spatio-temporal alignment module that allows each object to adaptively decode the optimal alignment proposal from multiple hypotheses without direct supervision. Specifically, HAT first utilizes multiple explicit motion models to generate spatial anchors and motion-aware feature proposals for historical instances. It then performs multi-hypothesis decoding by incorporating semantic and motion cues embedded in cached object queries, ultimately providing the optimal alignment proposal for the target frame. On nuScenes, HAT consistently improves 3D temporal detectors and trackers across diverse baselines. It achieves state-of-the-art tracking results with 46.0% AMOTA on the test set when paired with the DETR3D detector. In an object-centric E2E AD method, HAT enhances perception accuracy (+1.3% mAP, +3.1% AMOTA) and reduces the collision rate by 32%. When semantics are corrupted (nuScenes-C), the enhancement of motion modeling by HAT enables more robust perception and planning in the E2E AD.

Problem

• Adaptively aligns objects across frames for 3D perception
• Improves motion modeling in autonomous driving systems
• Enhances robustness in end-to-end perception and planning

Innovation

• Adaptive multi-hypothesis decoding for spatio-temporal alignment
• Utilizes multiple explicit motion models to generate proposals
• Incorporates semantic and motion cues without direct supervision