GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

2025-04-30

AI Summary

Current multimodal large language models (MLLMs) for document understanding lack fine-grained, attributable evaluation benchmarks, hindering precise identification of capability bottlenecks and systematic optimization. To address this, we propose GDI-Bench—the first fine-grained benchmark for general-purpose document intelligence—covering 9 real-world scenarios, 19 diverse tasks, and 1.9K document images. We introduce a novel decoupled evaluation paradigm that separately assesses visual perception and reasoning capabilities, enabling complexity-aware difficulty grading. Additionally, we design an intelligent knowledge-preserving training strategy to mitigate catastrophic forgetting during supervised fine-tuning (SFT). Leveraging GDI-Bench, we uncover significant visual perception deficiencies in state-of-the-art models such as GPT-4o. Our custom-built GDI model achieves SOTA performance on GDI-Bench and multiple established benchmarks, demonstrating both diagnostic utility and modeling advancement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 1.9k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate the GDI-Bench on various open-source and closed-source models, conducting decoupled analyses in the visual and reasoning domains. For instance, the GPT-4o model excels in reasoning tasks but exhibits limitations in visual capabilities. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI Model that mitigates the issue of catastrophic forgetting during the supervised fine-tuning (SFT) process through a intelligence-preserving training strategy. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and model will be open source.

Problem

• Need for comprehensive benchmark to evaluate multimodal document intelligence models
• Existing benchmarks lack systematic model weakness identification and improvement guidance
• Proposing GDI-Bench to decouple visual and reasoning complexities for graded assessment

Innovation

• Decouples visual and reasoning complexity for assessment
• Introduces intelligence-preserving training to prevent forgetting
• Features 1.9k images across 9 scenarios and 19 tasks