I'm Sorry Driver, I'm Afraid I Can't Do That: Appraising the Safety of LLMs within Automotive Contexts

2026-06-12

AI Summary

Current large language model (LLM) integration approaches struggle to meet real-time performance and functional safety requirements in automotive safety-critical scenarios. This work presents the first systematic mapping of LLM-related risks onto automotive safety standards, including ISO 21448 (SOTIF) and ISO/PAS 8800, establishing a safety-case-based analytical framework. Leveraging the Talk2Drive open-source case study, the research identifies key limitations such as latency, semantic misalignment, and hazardous event generation. The findings expose critical safety gaps in existing LLM deployment strategies and propose forward-looking assurance mechanisms tailored to LLM-specific risks. This study provides both theoretical grounding and practical pathways toward developing safe, trustworthy intelligent vehicle systems.

Abstract

This paper appraises recent frameworks within AI development to integrate LLMs into control tasks in automotive contexts from the perspective of safety assurance. This work has built upon the rapid integration of LLMs across automotive settings. However, we find that at present, these frameworks face significant challenges, limiting their efficacy in real-time safety-critical contexts. Firstly, we consider conceptual challenges, including the fact that deployers are faced with a dual challenge, wherein they must assure a model which has been developed upstream, i.e. as general-purpose tools by the large AI labs, in a downstream context, i.e. into specific vehicle architectures. Secondly, we consider concrete challenges from across existing standards. We show that there are currently both fundamental engineering constraints covered in ISO21448, such as latency, and novel LLM-specific issues, such as alignment-related issues covered in ISO/PAS8800. We ground both examples in a concrete introductory, experimental case study exploring an existing open-source repository, Talk2Drive. We present a safety argument in order to make explicit the limitations of existing solutions. Nonetheless, given that the use of LLMs in automotive contexts is being explored at a technical level and operationalised, we propose potential assurance mechanisms for LLM-related hazardous events going forward.

Problem

• LLM safety
• automotive systems
• safety assurance
• real-time constraints
• model alignment

Innovation

• LLM safety
• automotive AI
• safety assurance
• ISO 21448
• AI alignment