LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design

2026-07-02

AI Summary

This work addresses the degradation in perception robustness in autonomous driving caused by occlusion, adverse weather, and channel noise in vision–radar fusion. To this end, the authors propose LM-SCIP, a novel framework that, for the first time, leverages a large language model (LLM) as the reasoning hub for multimodal fusion. The architecture integrates a hierarchical vision–radar encoder, a Channel-Adaptive Semantic Module (CASM), and a Heterogeneous Mixture-of-Experts (H-MoE), employing a channel prompting mechanism to dynamically adjust fusion strategies. A LoRA-finetuned LLM drives a multi-task decoder to jointly output localization, trajectory prediction, and image reconstruction. Evaluated on nuScenes and VIRAT datasets, the method reduces localization RMSE by 40.0% over vision-only baselines under radar failure, achieving 0.214 m localization RMSE and 0.179 m minFDE (k=1), thereby significantly enhancing robustness and fusion gains across high and low signal-to-noise ratios.

Abstract

Vision-radar fusion is central to robust autonomous driving, combining dense visual semantics with precise range and velocity measurements from radar. However, real-world fusion quality is fundamentally challenged by dynamically varying input quality, stemming from occlusion, adverse weather, and channel noise. To address this, we re-frame the problem from static data fusion to channel-aware semantic reasoning and propose a Large Language Model-centric Semantic-layer Channel-aware Integrated Perception (LM-SCIP) framework. It places a Large Language Model (LLM) as a central reasoning core to fuse a local visual stream with a quality-varying external radar stream used to cover perception-blind spots. Concretely, LM-SCIP couples a hierarchical radar-vision encoder with a Channel-Adaptive Semantic Module (CASM) that maps link indicators into a "Channel Prompt" to dynamically gate external radar features. A parameter-efficient, LoRA-tuned LLM, in conjunction with a heterogeneous Mixture-of-Experts (H-MoE), then arbitrates between local visual cues and the channel-conditioned radar context. Finally, a decoupled multi-task decoder outputs localization, trajectory forecasting, and image reconstruction. Experiments on nuScenes and VIRAT validate our approach. On nuScenes, under a controlled toggle of radar input, LM-SCIP reduces localization RMSE by 40.0% versus a vision-only baseline. On VIRAT, the model attains a 0.214m localization RMSE and 0.179m minFDE (k=1). These results reveal that the proposed LM-SCIP enables a robust vision-dominant fallback at low SNR and synergistic fusion at high SNR.

Problem

• vision-radar fusion
• input quality variation
• channel-aware perception
• autonomous driving
• multimodal fusion

Innovation

• Large Language Model (LLM)
• Channel-Adaptive Fusion
• Semantic Reasoning
• Vision-Radar Integration
• Mixture-of-Experts (MoE)