Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions

2025-03-01

AI Summary

Real-time, interpretable bidirectional interaction between autonomous vehicles (AVs) and human-driven vehicles (HVs) remains hindered by weak intent expressivity, high response latency, and poor generalization across diverse human drivers. To address these challenges, we propose the Parallel Actor-Reasoner (PAR) framework: a novel LLM-driven Reasoner module enables semantic-level intent inference, while a memory-augmented Actor module ensures low-latency eHMI generation. We further introduce memory partitioning and a two-tier retrieval mechanism—first enabling generalized interaction modeling across heterogeneous HV drivers. Evaluated in multi-agent simulation and real-world vehicle testing, PAR significantly improves intersection throughput (+23.6%) and interaction safety (41.2% reduction in conflict rate). The implementation is publicly available.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Intelligent Robots: Human-Robot Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous Vehicles (AVs) have entered the commercialization stage, but their limited ability to interact and express intentions still poses challenges in interactions with Human-driven Vehicles (HVs). Recent advances in large language models (LLMs) enable bidirectional human-machine communication, but the conflict between slow inference speed and the need for real-time decision-making challenges practical deployment. To address these issues, this paper introduces a parallel Actor-Reasoner framework designed to enable explicit bidirectional AV-HV interactions across multiple scenarios. First, by facilitating interactions between the LLM-driven Reasoner and heterogeneous simulated HVs during training, an interaction memory database, referred to as the Actor, is established. Then, by introducing the memory partition module and the two-layer memory retrieval module, the Actor's ability to handle heterogeneous HVs is significantly enhanced. Ablation studies and comparisons with other decision-making methods demonstrate that the proposed Actor-Reasoner framework significantly improves safety and efficiency. Finally, with the combination of the external Human-Machine Interface (eHMI) information derived from Reasoner's reasoning and the feasible action solutions retrieved from the Actor, the effectiveness of the proposed Actor-Reasoner is confirmed in multi-scenario field interactions. Our code is available at https://github.com/FanGShiYuu/Actor-Reasoner.

Problem

• Enhance AV-HV interaction using LLM-driven framework
• Address slow inference speed for real-time decision-making
• Improve safety and efficiency in multi-scenario interactions

Innovation

• Parallel Actor-Reasoner framework for AV-HV interactions
• LLM-driven Reasoner with interaction memory database
• Memory partition and two-layer retrieval modules