To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software

2025-08-28

AI Summary

To address error accumulation and insufficient inter-task coordination in modular autonomous driving perception pipelines, this paper proposes a unified perception paradigm that jointly models detection, tracking, and motion prediction within a shared architecture—preserving interpretability while enhancing robustness, contextual reasoning, and computational efficiency. We introduce the first systematic taxonomy categorizing unified perception into early-, late-, and fully-unified paradigms, establishing the inaugural theoretical framework for the field. Through task integration analysis, trajectory modeling, and representation flow characterization—combined with multimodal architectural surveys across mainstream benchmarks—we provide a comprehensive review. Furthermore, we consolidate open-source resources and technical roadmaps to delineate concrete future research directions. This work lays foundational groundwork for developing next-generation perception systems that are efficient, generalizable, and interpretable.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Autonomous vehicle perception typically relies on modular pipelines that decompose the task into detection, tracking, and prediction. While interpretable, these pipelines suffer from error accumulation and limited inter-task synergy. Unified perception has emerged as a promising paradigm that integrates these sub-tasks within a shared architecture, potentially improving robustness, contextual reasoning, and efficiency while retaining interpretable outputs. In this survey, we provide a comprehensive overview of unified perception, introducing a holistic and systemic taxonomy that categorizes methods along task integration, tracking formulation, and representation flow. We define three paradigms -Early, Late, and Full Unified Perception- and systematically review existing methods, their architectures, training strategies, datasets used, and open-source availability, while highlighting future research directions. This work establishes the first comprehensive framework for understanding and advancing unified perception, consolidates fragmented efforts, and guides future research toward more robust, generalizable, and interpretable perception.

Problem

• Addressing error accumulation in modular autonomous vehicle perception pipelines
• Integrating detection, tracking and prediction into unified perception architectures
• Establishing comprehensive taxonomy and framework for unified perception methods

Innovation

• Unified perception integrating detection tracking prediction
• Shared architecture improving robustness contextual reasoning
• Taxonomy categorizing methods task integration tracking formulation