When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network

2025-06-20

AI Summary

To address the fundamental trade-off between latency and accuracy in anomaly detection for autonomous driving, this paper proposes a real-time-oriented multimodal asynchronous hybrid architecture. Methodologically, it introduces the first asynchronous graph neural network (GNN)–CNN collaborative framework that synergistically fuses high-temporal-resolution event streams from event cameras with spatially rich RGB image features. Key innovations include event-driven dynamic graph construction, cross-modal feature alignment, and a lightweight fusion mechanism. The core contribution lies in breaking the conventional latency–accuracy Pareto frontier: on standard benchmarks, the method achieves an end-to-end inference latency of <15 ms while improving mean average precision (mAP) by 4.2% over prior real-time methods—setting a new state-of-the-art for real-time anomaly detection. This enables millisecond-level, high-confidence anomaly responses essential for safety-critical autonomous systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Anomaly detection is essential for the safety and reliability of autonomous driving systems. Current methods often focus on detection accuracy but neglect response time, which is critical in time-sensitive driving scenarios. In this paper, we introduce real-time anomaly detection for autonomous driving, prioritizing both minimal response time and high accuracy. We propose a novel multimodal asynchronous hybrid network that combines event streams from event cameras with image data from RGB cameras. Our network utilizes the high temporal resolution of event cameras through an asynchronous Graph Neural Network and integrates it with spatial features extracted by a CNN from RGB images. This combination effectively captures both the temporal dynamics and spatial details of the driving environment, enabling swift and precise anomaly detection. Extensive experiments on benchmark datasets show that our approach outperforms existing methods in both accuracy and response time, achieving millisecond-level real-time performance.

Problem

• Real-time anomaly detection for autonomous driving systems
• Balancing minimal response time and high detection accuracy
• Integrating event and RGB camera data for improved performance

Innovation

• Multimodal asynchronous hybrid network for anomaly detection
• Combines event streams with RGB images
• Achieves millisecond-level real-time performance