LLM-MLFFN: Multi-Level Autonomous Driving Behavior Feature Fusion via Large Language Model

2026-03-02

AI Summary

This work addresses the limitations of existing autonomous driving behavior classification methods, which rely solely on numerical time-series modeling and lack semantic abstraction, resulting in poor interpretability and robustness in complex traffic scenarios. To overcome this, the authors propose the LLM-MLFFN framework, which integrates large language models (LLMs) into the task for the first time. The framework employs a dual-channel, multi-level fusion network to jointly model numerical features—encompassing statistical, behavioral, and dynamic aspects—with semantic descriptions generated by an LLM. A weighted attention mechanism enables adaptive feature fusion. Evaluated on the Waymo Open Motion Dataset, the method achieves over 94% accuracy in behavior classification, significantly outperforming current approaches. Ablation studies confirm the contribution of each component, demonstrating that the framework delivers high accuracy, strong robustness, and enhanced interpretability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Accurate classification of autonomous vehicle (AV) driving behaviors is critical for safety validation, performance diagnosis, and traffic integration analysis. However, existing approaches primarily rely on numerical time-series modeling and often lack semantic abstraction, limiting interpretability and robustness in complex traffic environments. This paper presents LLM-MLFFN, a novel large language model (LLM)-enhanced multi-level feature fusion network designed to address the complexities of multi-dimensional driving data. The proposed LLM-MLFFN framework integrates priors from largescale pre-trained models and employs a multi-level approach to enhance classification accuracy. LLM-MLFFN comprises three core components: (1) a multi-level feature extraction module that extracts statistical, behavioral, and dynamic features to capture the quantitative aspects of driving behaviors; (2) a semantic description module that leverages LLMs to transform raw data into high-level semantic features; and (3) a dual-channel multi-level feature fusion network that combines numerical and semantic features using weighted attention mechanisms to improve robustness and prediction accuracy. Evaluation on the Waymo open trajectory dataset demonstrates the superior performance of the proposed LLM-MLFFN, achieving a classification accuracy of over 94%, surpassing existing machine learning models. Ablation studies further validate the critical contributions of multi-level fusion, feature extraction strategies, and LLM-derived semantic reasoning. These results suggest that integrating structured feature modeling with language-driven semantic abstraction provides a principled and interpretable pathway for robust autonomous driving behavior classification.

Problem

• autonomous driving behavior classification
• semantic abstraction
• time-series modeling
• interpretability
• robustness

Innovation

• Large Language Model
• Multi-level Feature Fusion
• Autonomous Driving Behavior Classification
• Semantic Abstraction
• Dual-channel Attention