KLDrive: Fine-Grained 3D Scene Reasoning for Autonomous Driving based on Knowledge Graph

2026-03-21

AI Summary

This work addresses key limitations in fine-grained 3D scene reasoning for autonomous driving—namely, unreliable factual grounding, hallucination, opaque reasoning processes, and heavy reliance on task-specific training. To overcome these challenges, the authors propose KLDrive, the first reasoning framework that integrates knowledge graphs with large language models (LLMs). KLDrive constructs a reliable scene knowledge graph via an energy-based formulation and guides the LLM toward factually grounded, interpretable reasoning under structured constraints, without requiring extensive task-specific fine-tuning. Core technical components include multi-source evidence fusion, structured prompting, few-shot exemplars, and constrained action-space reasoning. On the NuScenes-QA and GVQA benchmarks, KLDrive achieves 65.04% accuracy and a 42.45 SPICE score, respectively, outperforming the strongest baseline by 46.01 percentage points on the most challenging counting tasks.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.

Problem

• autonomous driving
• fine-grained 3D scene reasoning
• scene fact hallucination
• knowledge graph
• question answering

Innovation

• knowledge graph
• large language model
• scene reasoning
• autonomous driving
• fact grounding