🤖 AI Summary
This work addresses the lack of reproducible training methodologies in existing long-context vision-language models, which hinders effective visual question answering over lengthy documents. The study systematically investigates efficient training strategies—combining continual pretraining, supervised fine-tuning, and preference optimization—for models at 24B and 32B parameter scales, enabling visual document understanding with context lengths up to 344K tokens. Key innovations include aligning training sequence lengths with evaluation settings for improved efficacy, introducing page-index embeddings to significantly boost performance, leveraging synthetic data for self-augmented training, and demonstrating for the first time that long-context visual training can enhance pure textual long-context capabilities. The approach achieves state-of-the-art results on MMLongBenchDoc and releases a refined, high-quality benchmark, MMLBD-C, to support future research.
📝 Abstract
We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text. While several such strong are open-weight, namely Qwen3 VL and GLM 4.5/6V, their training recipes and data pipelines are not reproducible. We systematically study continued pretraining, supervised finetuning, and preference optimization for 24B and 32B parameter models, backed by extensive LC evaluations and ablations to bridge this gap, and achieve state-of-the-art performance on MMLongBenchDoc for both parameter scales. In addition to this, our key findings include: (i) training on context lengths that match evaluation context lengths outperforms training on longer contexts, (ii) training and evaluating with page indices provides a simple, high-impact boost to long-document performance, (iii) our synthetic data pipelines enable self-improvement via continued pretraining and supervised finetuning, and (iv) we extend the known text-to-visual long context transfer to the reverse, showing that visual long context training transfers to long-context text performance. We also release MMLBD-C, a manually corrected version of MMLongBenchDoc to reduce erroneous and low quality examples in the benchmark.