Enhancing Large Vision-Language Models with Layout Modality for Table Question Answering on Japanese Annual Securities Reports

2025-05-23

AI Summary

Existing large vision-language models (LVLMs) exhibit significant limitations in jointly understanding the textual semantics and spatial layout of tables in Japanese securities annual reports, failing to preserve structural information end-to-end. To address this, we propose the first LVLM framework that jointly models fine-grained intra-table textual semantics and explicit layout features—including row/column indices and relative coordinates. Our approach extracts text sequences via OCR, injects layout modality using relative position encoding, and introduces a text-image-layout ternary cross-attention mechanism for multimodal alignment. Crucially, it operates directly on raw inputs without pre-formatting. On Japanese annual report table question answering, our method achieves a 12.7% absolute accuracy gain over strong baselines, substantially outperforming text-only and image-only approaches. Moreover, it demonstrates robust performance across diverse table formats—including HTML-rendered tables, screenshots, and scanned documents—highlighting its practical applicability in real-world financial document analysis.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

With recent advancements in Large Language Models (LLMs) and growing interest in retrieval-augmented generation (RAG), the ability to understand table structures has become increasingly important. This is especially critical in financial domains such as securities reports, where highly accurate question answering (QA) over tables is required. However, tables exist in various formats-including HTML, images, and plain text-making it difficult to preserve and extract structural information. Therefore, multimodal LLMs are essential for robust and general-purpose table understanding. Despite their promise, current Large Vision-Language Models (LVLMs), which are major representatives of multimodal LLMs, still face challenges in accurately understanding characters and their spatial relationships within documents. In this study, we propose a method to enhance LVLM-based table understanding by incorporating in-table textual content and layout features. Experimental results demonstrate that these auxiliary modalities significantly improve performance, enabling robust interpretation of complex document layouts without relying on explicitly structured input formats.

Problem

• Enhancing table understanding in Japanese financial reports using layout modality
• Improving accuracy of table question answering with multimodal LLMs
• Addressing challenges in character and spatial relationship understanding in documents

Innovation

• Enhance LVLMs with layout modality
• Incorporate in-table text and layout
• Improve table understanding accuracy