Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution

2026-08-08

AI Summary

Understanding visually rich multi-page documents poses significant challenges due to sparse, cross-page distributed evidence that often exceeds model context windows. This work systematically attributes performance bottlenecks to three error modes—representation, selection, and reasoning—and quantifies the impact of each through modular intervention experiments, wherein two components are fixed while one is perturbed. The study reveals that although visual information is essential, it cannot substitute for textual content; omitting critical pages substantially degrades accuracy, whereas irrelevant pages have limited effect. Furthermore, current reasoning modules struggle to effectively integrate cross-page evidence, yet their performance can be markedly improved through carefully designed prompts.

Abstract

Multi-page visually-rich document understanding (MP-VRDU) requires managing evidence that is sparse, spread across pages, and often exceeds a model's context window. Prior work has produced competing, largely untested claims about how these systems should be built. We attribute incorrect answers to three failure modes, representation, selection, and reasoning, and isolate each over a multi-page document understanding dataset by intervening on one while holding the others fixed. We find that vision is necessary but does not replace text extraction, that missing pages bound accuracy while distractors cost little, and that reasoners fail to integrate evidence across pages even when it is fully supplied. Prompting can shift reasoning behaviour substantially, improving some outcomes at the expense of others. We translate these findings into guidance for building such systems under a fixed compute budget.

Problem

• multi-page document understanding
• visually-rich documents
• failure attribution
• evidence integration
• context window limitation

Innovation

• multi-page document understanding
• failure attribution
• vision-language reasoning
• context window limitation
• prompting intervention