🤖 AI Summary
Current autonomous driving systems lack interpretable, behavior-level decision descriptions, which hinders their safety, trustworthiness, and regulatory compliance. This work proposes a multimodal large language model framework that, for the first time, integrates quantized LoRA-finetuned LLMs with Q-Former adapters to efficiently map spatiotemporal fusion of LiDAR and multi-camera bird’s-eye-view inputs into human-readable behavioral semantics. Evaluated on the newly curated CommandLM-nuScenes dataset, the approach substantially outperforms the BLIP-2 baseline, achieving CIDEr and BERT-F1 scores of 0.67 and 0.88, respectively. Human evaluation confirms that 58% of the generated descriptions are accurate, concise, and compliant with driving norms.
📝 Abstract
As autonomous driving systems move toward real-world deployment, interpretable, behavior-level decision-making is essential for safety, trust, and regulation. We introduce CommandLM, a multimodal large language model that generates concise, human-readable behavior descriptions for ego vehicles from fused multi-sensor data. Our model processes temporally fused bird's-eye view representations from LiDAR and multi-camera inputs via a Q-Former adapter connected to a quantized, LoRA-fine-tuned large language model. Trained on our CommandLM-nuScenes dataset, CommandLM produces intent-aware, interpretable captions suitable for planner supervision and safety auditing. Experiments demonstrate strong linguistic and behavioral alignment, achieving CIDEr 0.67, and BERT-F1 0.88, substantially outperforming the BLIP-2 baseline (CIDEr 0.52, BERT-F1 0.86). In human evaluation, 58% of the generated descriptions were rated accurate, efficient and rule-compliant, confirming their real-world plausibility. While the remaining descriptions may not always select the most efficient, goal-oriented behavior, CommandLM's interpretable outputs enable downstream validation systems to identify and correct such cases, making it an effective tool for transparent behavior auditing. These results show that integrating multimodal fusion with language reasoning yields efficient and transparent behavior-level understanding for autonomous driving. We release our code and dataset at: https://github.com/b-tok/CommandLM