Foundation Models for Autonomous Driving System: An Initial Roadmap

2025-04-01

AI Summary

Foundational models (FMs) face significant challenges in integration, verification, and deployment within autonomous driving systems (ADS), primarily due to safety-critical constraints and the lack of systematic frameworks. Method: This paper proposes the first FM-oriented, systematic integration framework for ADS—spanning FM infrastructure construction, scenario-specific application adaptation, and industrial-grade implementation pathways. It synergistically integrates large language models, multimodal perception modeling, formal verification, software engineering principles, and system-level evaluation paradigms, and identifies, for the first time, key interdisciplinary challenges and research gaps in FM–safety-critical automotive system co-design. Contribution: We establish a scalable, verifiable FM-driven ADS technology roadmap covering architecture design, functional verification, and safety assurance across the full development lifecycle. This bridges critical theoretical and practical gaps in integrating FMs with functionally safe, high-assurance vehicular systems, providing a unified R&D guideline for both academia and industry.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Other Foundations of Multi Agent Systems
• Application Domains: Mobility, Driving & Flight

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Recent advancements in Foundation Models (FMs), such as Large Language Models (LLMs), have significantly enhanced Autonomous Driving Systems (ADSs) by improving perception, reasoning, and decision-making in dynamic and uncertain environments. However, ADSs are highly complex cyber-physical systems that demand rigorous software engineering practices to ensure reliability and safety. Integrating FMs into ADSs introduces new challenges in system design and evaluation, requiring a systematic review to establish a clear research roadmap. To unlock these challenges, we present a structured roadmap for integrating FMs into autonomous driving, covering three key aspects: the infrastructure of FMs, their application in autonomous driving systems, and their current applications in practice. For each aspect, we review the current research progress, identify existing challenges, and highlight research gaps that need to be addressed by the community.

Problem

• Integrating Foundation Models into Autonomous Driving Systems
• Addressing reliability and safety challenges in FM-ADS integration
• Establishing a research roadmap for FM applications in ADS

Innovation

• Integrating Foundation Models into autonomous driving
• Systematic review of FM infrastructure and applications
• Addressing challenges in perception, reasoning, decision-making