From Prompts to Pavement: LMMs-based Agentic Behavior-Tree Generation Framework for Autonomous Vehicles

2026-01-18

AI Summary

This work addresses the limitations of traditional behavior trees in autonomous driving, whose static structures and reliance on manual tuning hinder adaptability in complex, dynamic environments—particularly impeding progress toward Level 5 autonomy. To overcome this, the authors propose an agent framework that integrates a large language model (LLM) with a multimodal vision model (LVM). When the baseline behavior tree fails, the system dynamically generates executable behavior subtrees through chain-of-symbol prompting and in-context learning, enabling adaptive decision-making without human intervention. This approach represents the first integration of chain-of-symbol prompting with behavior tree generation. Extensive experiments on the CARLA+Nav2 simulation platform demonstrate its effectiveness and generalization capability in handling unforeseen scenarios such as street blockages.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous vehicles (AVs) require adaptive behavior planners to navigate unpredictable, real-world environments safely. Traditional behavior trees (BTs) offer structured decision logic but are inherently static and demand labor-intensive manual tuning, limiting their applicability at SAE Level 5 autonomy. This paper presents an agentic framework that leverages large language models (LLMs) and multi-modal vision models (LVMs) to generate and adapt BTs on the fly. A specialized Descriptor agent applies chain-of-symbols prompting to assess scene criticality, a Planner agent constructs high-level sub-goals via in-context learning, and a Generator agent synthesizes executable BT sub-trees in XML format. Integrated into a CARLA+Nav2 simulation, our system triggers only upon baseline BT failure, demonstrating successful navigation around unexpected obstacles (e.g., street blockage) with no human intervention. Compared to a static BT baseline, this approach is a proof-of-concept that extends to diverse driving scenarios.

Problem

• Autonomous Vehicles
• Behavior Trees
• Adaptive Planning
• SAE Level 5 Autonomy
• Real-world Navigation

Innovation

• Behavior Tree
• Large Language Models
• Autonomous Vehicles
• Agentic Framework
• Multimodal Vision Models