RESPOND: Risk-Enhanced Structured Pattern for LLM-driven Online Node-level Decision-making

2025-12-23

AI Summary

To address inaccurate scene retrieval and inefficient reflection caused by unstructured textual memory in LLM-driven autonomous driving, this paper proposes an explicit risk-pattern-based structured decision-making framework. Methodologically, it introduces: (1) a novel 5×3 spatial risk matrix for structured driving scene representation; (2) a two-tier memory mechanism coupled with a pattern-aware reflection module, enabling “one-collision generalization” and adaptive decision-making across high- and low-risk scenarios; and (3) structured scene encoding, a hybrid rule–LLM decision pipeline, and sub-pattern matching for precise risk identification. Evaluation on highway-env shows a significant reduction in collision rate; the system acquires a Sporty driving style via human feedback within 20 steps; and in HighD high-risk cut-in scenarios, it outperforms human intervention in risk mitigation for 84.9% of cases.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Machine Learning: Structured Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Current LLM-based driving agents that rely on unstructured plain-text memory suffer from low-precision scene retrieval and inefficient reflection. To address this limitation, we present RESPOND, a structured decision-making framework for LLM-driven agents grounded in explicit risk patterns. RESPOND represents each ego-centric scene using a unified 5 by 3 matrix that encodes spatial topology and road constraints, enabling consistent and reliable retrieval of spatial risk configurations. Based on this representation, a hybrid rule and LLM decision pipeline is developed with a two-tier memory mechanism. In high-risk contexts, exact pattern matching enables rapid and safe reuse of verified actions, while in low-risk contexts, sub-pattern matching supports personalized driving style adaptation. In addition, a pattern-aware reflection mechanism abstracts tactical corrections from crash and near-miss frames to update structured memory, achieving one-crash-to-generalize learning. Extensive experiments demonstrate the effectiveness of RESPOND. In highway-env, RESPOND outperforms state-of-the-art LLM-based and reinforcement learning based driving agents while producing substantially fewer collisions. With step-wise human feedback, the agent acquires a Sporty driving style within approximately 20 decision steps through sub-pattern abstraction. For real-world validation, RESPOND is evaluated on 53 high-risk cut-in scenarios extracted from the HighD dataset. For each event, intervention is applied immediately before the cut-in and RESPOND re-decides the driving action. Compared to recorded human behavior, RESPOND reduces subsequent risk in 84.9 percent of scenarios, demonstrating its practical feasibility under real-world driving conditions. These results highlight RESPONDs potential for autonomous driving, personalized driving assistance, and proactive hazard mitigation.

Problem

• Addresses low-precision scene retrieval in LLM-based driving agents
• Enables safe action reuse in high-risk contexts via pattern matching
• Reduces collision risk through structured memory and reflection mechanisms

Innovation

• Structured 5x3 matrix encodes spatial topology and constraints
• Hybrid rule and LLM pipeline with two-tier memory mechanism
• Pattern-aware reflection abstracts corrections for one-crash-to-generalize learning