Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Prediction

2026-04-23

AI Summary

This work proposes a general-purpose trajectory prediction framework based on frozen large language models (LLMs) to enhance spatiotemporal reasoning in autonomous driving by integrating dynamic traffic participant behaviors with static high-definition map semantics. The approach employs a traffic encoder and a lightweight CNN to extract dynamic scene features and road topology, respectively, and uses reprogramming adapters to translate these representations into LLM-comprehensible tokens. A linear decoder then generates multimodal future trajectories. This study presents the first systematic evaluation of frozen LLMs for trajectory prediction under fused map and traffic inputs, demonstrating the critical role of map semantics and showing strong generalization capabilities and low adaptation costs across diverse LLM architectures.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have recently demonstrated strong reasoning capabilities and attracted increasing research attention in the field of autonomous driving (AD). However, safe application of LLMs on AD perception and prediction still requires a thorough understanding of both the dynamic traffic agents and the static road infrastructure. To this end, this study introduces a framework to evaluate the capability of LLMs in understanding the behaviors of dynamic traffic agents and the topology of road networks. The framework leverages frozen LLMs as the reasoning engine, employing a traffic encoder to extract spatial-level scene features from observed trajectories of agents, while a lightweight Convolutional Neural Network (CNN) encodes the local high-definition (HD) maps. To assess the intrinsic reasoning ability of LLMs, the extracted scene features are then transformed into LLM-compatible tokens via a reprogramming adapter. By residing the prediction burden with the LLMs, a simpler linear decoder is applied to output future trajectories. The framework enables a quantitative analysis of the influence of multi-modal information, especially the impact of map semantics on trajectory prediction accuracy, and allows seamless integration of frozen LLMs with minimal adaptation, thereby demonstrating strong generalizability across diverse LLM architectures and providing a unified platform for model evaluation.

Problem

• trajectory prediction
• large language models
• map-aware reasoning
• spatio-temporal reasoning
• autonomous driving

Innovation

• Frozen LLMs
• Map-aware reasoning
• Trajectory prediction
• Reprogramming adapter
• Spatio-temporal reasoning