SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

2026-07-11

AI Summary

Existing vision-language models exhibit unclear failure modes in real-world document understanding due to the strong coupling among document length, layout, modality, and question type. To disentangle these factors, this work introduces the first fully synthetic benchmark for long-context visual document understanding, enabling independent control over each factor through combinatorial design and incorporating 40% random perturbations to eliminate spurious correlations. The dataset comprises highly diverse long documents generated end-to-end by large language models across six canonical layout templates, following a systematic experimental design. Evaluation across seven state-of-the-art models reveals three critical failure patterns previously undetectable with existing benchmarks: sharp performance degradation with increasing document length, weakest comprehension of content in the middle sections of documents, and a complete breakdown in chart understanding within long documents.

Abstract

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

Problem

• visual document understanding
• long-context
• benchmark
• vision language models
• controlled evaluation

Innovation

• SynthDocBench
• controlled benchmark
• long-context visual document understanding
• combinatorial design
• vision-language models