Robust sensor fusion against on-vehicle sensor staleness

2025-06-06

AI Summary

In autonomous driving, trajectory prediction reliability is severely degraded by sensor staleness—temporal misalignment among multi-sensor data caused by transmission delays. To address this, we propose a robust asynchronous multimodal fusion framework. Our method introduces point-level temporal offset features to explicitly model the time lag of LiDAR and radar relative to camera inputs; designs a vehicle-observed stale-pattern-based data augmentation strategy that is model-agnostic and plug-and-play; and integrates perspective-domain detection, cross-modal temporal alignment, temporal-aware feature encoding, and simulation-enhanced training. Evaluated under both synchronized and diverse stale conditions—including severe unimodal latency—the framework demonstrates stable, state-of-the-art performance, significantly outperforming baselines while maintaining safety-critical trajectory prediction accuracy even under extreme sensor delay.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Sensor fusion is crucial for a performant and robust Perception system in autonomous vehicles, but sensor staleness, where data from different sensors arrives with varying delays, poses significant challenges. Temporal misalignment between sensor modalities leads to inconsistent object state estimates, severely degrading the quality of trajectory predictions that are critical for safety. We present a novel and model-agnostic approach to address this problem via (1) a per-point timestamp offset feature (for LiDAR and radar both relative to camera) that enables fine-grained temporal awareness in sensor fusion, and (2) a data augmentation strategy that simulates realistic sensor staleness patterns observed in deployed vehicles. Our method is integrated into a perspective-view detection model that consumes sensor data from multiple LiDARs, radars and cameras. We demonstrate that while a conventional model shows significant regressions when one sensor modality is stale, our approach reaches consistently good performance across both synchronized and stale conditions.

Problem

• Addressing sensor staleness in autonomous vehicle perception
• Resolving temporal misalignment in multi-sensor data fusion
• Improving trajectory prediction quality despite delayed sensor inputs

Innovation

• Per-point timestamp offset feature for temporal alignment
• Data augmentation simulating realistic sensor staleness
• Model-agnostic approach for robust sensor fusion