DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

2026-05-25

AI Summary

This work addresses the challenge that lightweight vision-language models struggle to jointly localize multiple objects, attributes, and relations in dense visual scenes and perform multi-step reasoning. To overcome this limitation without altering model architecture, the authors propose DRScaffold, a framework that enforces visually grounded, structured reasoning through causally ordered supervision across four stages. Additionally, they introduce DRBench, the first benchmark specifically designed for evaluating reasoning in dense scenes. Through supervised fine-tuning, DRScaffold substantially enhances the performance of lightweight models such as Qwen2.5-VL-3B on DRBench—surpassing even frozen 32B-scale models—while preserving their general-purpose capabilities.

Abstract

Lightweight vision-language models perform competitively on standard benchmarks yet fail systematically in dense-scene reasoning, where multiple objects, attributes, and relations must be jointly grounded and resolved through multi-step inference. Such capability is critical for real-world applications where models must reliably interpret cluttered environments. Yet existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations, leaving lightweight models free to generate fluent but visually unanchored reasoning chains. To address this gap, we first introduce DRBench, a benchmark of 14,573 questions across 2,943 images, organized into five task categories spanning three progressive reasoning layers. Building on DRBench, we propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages, enforcing grounded reasoning without architectural modification. Experiments on three lightweight VLMs demonstrate substantial gains on DRBench while preserving or improving performance on general-purpose benchmarks. Notably, Qwen2.5-VL-3B trained with DRScaffold surpasses the frozen Qwen2.5-VL-32B on DRBench, demonstrating that structured supervision can substitute for a significant portion of model scale in dense-scene reasoning. Our code and models are available at https://github.com/irene-shi/DRScaffold .

Problem

• dense-scene reasoning
• lightweight vision-language models
• visual grounding
• multi-step inference
• cluttered environments

Innovation

• dense-scene reasoning
• grounded reasoning
• lightweight vision-language models
• supervised fine-tuning
• DRBench