HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation

2026-08-14

AI Summary

This study addresses evidence misalignment and generation unfaithfulness in existing multimodal RAG systems caused by document flattening. We propose a Hierarchy-Aware Multimodal RAG framework that uniquely leverages document hierarchy as the core anchor for both retrieval and generation, employing a local image-text alignment mechanism to preserve the structural logic of source documents. Furthermore, we introduce HAM-Bench to validate the critical role of hierarchy awareness. Experimental results demonstrate that our approach improves average multimodal performance by 17.3% and achieves a 24.2% increase in Img-CBS on the Wukong dataset. These findings indicate significant enhancements in evidence alignment and generation faithfulness for structured document question answering, effectively mitigating the limitations inherent in conventional flattening-based approaches.

Abstract

Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git.

Problem

• Multimodal RAG
• Structure-Faithful Generation
• Document Hierarchy
• Interleaved Generation
• Local Text-Image Alignment

Innovation

• Hierarchy-Aware Multimodal RAG
• Structure-Faithful Generation
• Document Hierarchy Grounding
• Interleaved Generation
• HAM-Bench