AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning

2026-04-20

AI Summary

This work addresses the limitations of multimodal large language models in structured semantic tasks—such as small object recognition, counting, coarse localization, and relational reasoning—in aerial imagery. To overcome these challenges, the authors propose AeroRAG, a framework that transforms input images into structured visual knowledge encompassing object categories, counts, spatial locations, and semantic relationships. AeroRAG leverages scene graphs to generate intermediate representations and retrieves relevant semantic fragments to construct compact prompts for reasoning by a text-only large language model. Experimental results demonstrate that AeroRAG significantly outperforms six state-of-the-art multimodal baselines on both the AUG aerial dataset and VG-150, with particularly notable gains in dense scenes and relation-sensitive tasks. Furthermore, its generalizability is validated through strong performance on VQAv2.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite recent progress in multimodal large language models (MLLMs), reliable visual question answering in aerial scenes remains challenging. In such scenes, task-critical evidence is often carried by small objects, explicit quantities, coarse locations, and inter-object relations, whereas conventional dense visual-token representations are not well aligned with these structured semantics. To address this interface mismatch, we propose AeroRAG, a scene-graph-guided multimodal retrieval-augmented generation framework for visual question answering. The framework first converts an input image into structured visual knowledge, including object categories, quantities, spatial locations, and semantic relations, and then retrieves query-relevant semantic chunks to construct compact prompts for a text-based large language model. Rather than relying on direct reasoning over dense visual tokens, our method introduces a more explicit intermediate interface between perception and language reasoning. Experiments on the AUG aerial dataset and the general-domain VG-150 benchmark show consistent improvements over six strong MLLM baselines, with the largest gains observed in dense aerial scenes and relation-sensitive reasoning. We further evaluate the framework on VQAv2 to verify that the proposed interface remains compatible with standard visual reasoning settings. These results suggest that structured retrieval is a practical design direction for deployment-oriented and grounded visual reasoning systems.

Problem

• aerial visual reasoning
• multimodal large language models
• structured semantics
• visual question answering
• scene understanding

Innovation

• structured retrieval
• scene graph
• multimodal RAG
• aerial visual reasoning
• visual question answering