StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

2026-07-01

AI Summary

This work addresses the mismatch between multi-turn training and single-turn evaluation in visual instruction tuning, which often leads to visual attention decay and contextual overfitting. To bridge this structural gap between training and testing, the authors propose StochasT (Stochastic Turn Depth), a strategy that dynamically aggregates multiple language tasks associated with the same image into dialogues of varying lengths during training, while preserving the original task order. Additionally, they introduce an evaluation protocol based on balanced Latin squares to ensure fairness and comprehensiveness. Without discarding any data, this approach significantly enhances the unified performance, robustness, and multimodal reasoning coherence of large vision-language models across both single-turn and multi-turn scenarios.

Abstract

Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.

Problem

• Visual Instruction Tuning
• Multi-turn Training
• Attention Decay
• Contextual Overfitting
• Evaluation Mismatch

Innovation

• Stochastic Turn Depth
• Visual Instruction Tuning
• Multimodal Reasoning
• Contextual Robustness
• Balanced Latin Square