🤖 AI Summary
This work addresses the lack of effective internalized reasoning mechanisms in existing visual long-document understanding methods by introducing, for the first time, an approach that embeds reasoning capabilities directly into vision-language models. The authors construct a synthetic data pipeline to generate reasoning traces comprising page relevance scores, evidence extraction, and ranking. These traces are integrated into the model through supervised fine-tuning (SFT), augmented with <think> tags and <cot> control tokens, alongside a low-intensity model merging strategy. Evaluated on MMLongBenchDoc, the resulting Qwen3-VL-32B achieves a score of 58.3—surpassing the performance of Qwen3-VL-235B, which has over seven times more parameters—while a Mistral-based variant reduces reasoning output length by 12.4× and simultaneously improves performance by 3.8 points.
📝 Abstract
Visual long-document understanding is critical for enterprise, legal, and scientific applications, yet the best performing open recipes have not explored reasoning, a capability which has driven leaps in math and code performance. We introduce a synthetic data pipeline for reasoning in long-document understanding that generates thinking traces by scoring each page for question relevance, extracting textual evidence and ordering it from most to least relevant. We apply SFT to the resulting traces within \texttt{<think>} tags, gated by a \texttt{<cot>} control token, and the resulting reasoning capability is internalized via low-strength model merging. We study Qwen3 VL 32B and Mistral Small 3.1 24B. With Qwen3 VL, we achieve 58.3 on MMLongBenchDoc, surpassing the 7$\times$ larger Qwen3 VL 235B A22B (57.0). With Mistral, we show that synthetic reasoning outperforms distillation from the Thinking version's traces by 3.8 points on MMLBD-C, and internalized reasoning exhibits 12.4$\times$ fewer mean output tokens compared to explicit reasoning. We release our pipeline for reproducibility and further exploration.