PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs

2026-05-22

AI Summary

This work addresses the challenge of fine-grained visual understanding in multimodal large language models, which often struggle with precise localization due to insufficient supervisory signals. The authors propose a Programmatic Generation Task (PGT) framework that overlays unambiguous geometric primitives onto images to construct dense supervision, effectively decoupling visual localization from semantic priors and thereby enhancing fine-grained perception. PGT is the first approach to leverage programmatic synthetic tasks both to improve localization accuracy and to diagnose the root causes of perceptual failures at low cost, revealing that spatial reasoning bottlenecks stem primarily from data supervision rather than model architecture or resolution. Experiments demonstrate performance gains of up to 20% on What'sUp and 13.3% on CV-Bench-2D, with consistent improvements of +5.5% to +8.3% on state-of-the-art models while preserving general perceptual capabilities.

Abstract

Despite remarkable progress in Multimodal Large Language Models (MLLMs), these models still struggle with fine-grained understanding tasks. In this work, we propose Procedurally Generated Tasks (PGT), a simple data-driven framework that serves a dual purpose: inducing fine-grained visual understanding and acting as a low-cost diagnostic tool to identify the source of perception failures. By overlaying unambiguous geometric primitives on images, PGT generate additional dense supervision that disentangles visual grounding capability from semantic priors. Extensive experiments on relational, quantitative, and 3D/depth understanding benchmarks show that PGT yields remarkable gains across diverse architectures. Instruction tuning MLLMs on LLaVA-v1.5-Instruct augmented with PGT data results in improvements of up to +20% on the What'sUp benchmark and +13.3% on CV-Bench-2D, while maintaining general perception capabilities. Moreover, finetuning state-of-the-art MLLMs on PGT data leads to boosts of up to +5.5% on What'sUp and +8.3% on CV-Bench-2D. These findings demonstrate that PGT effectively address the bottleneck of fine-grained perception, revealing that many spatial reasoning deficits stem from inadequate supervision signals rather than inherent architectural or resolution limitations.

Problem

• visual grounding
• fine-grained understanding
• multimodal large language models
• spatial reasoning
• perception failures

Innovation

• Procedurally Generated Tasks
• visual grounding
• multimodal large language models
• fine-grained perception
• dense supervision