Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models

2026-06-18

AI Summary

This work addresses the challenge that multimodal large language models struggle to precisely align textual references with specific image regions in fine-grained spatial reasoning due to the absence of explicit geometric anchoring mechanisms. To overcome this limitation, the authors propose the Timage paradigm, which reframes multimodal understanding as an input-side alignment problem by generating typed textual overlays directly on the image to guide the model’s attention toward relevant visual regions. The approach leverages constrained Schrödinger Bridges (cSB) and optimizes layout in two stages: region search preserves foreground content, while appearance shaping balances text legibility and visual harmony. Key technical innovations include entropy-regularized optimal transport sampling, hard occlusion barriers, and an “ink budget” regularization. Evaluated on the VMCBench benchmark, the method achieves state-of-the-art performance using only a 7B-parameter backbone, significantly outperforming larger closed-source systems and parameter-efficient fine-tuning baselines.

Abstract

Multimodal Large Language Models (MLLMs) often lose track of the right image regions during fine-grained spatial reasoning, because a textual query rarely carries any explicit geometric anchor into the pixel domain. Prevailing remedies either rewire the model's weights or pad the prompt with verbose instructions, yet neither reliably pins the language to the correct visual coordinates without eroding the backbone's general competence. We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input: the query is drawn, as a typeset overlay, onto the image itself. The placement and appearance of this overlay are produced by a Constrained Schrödinger Bridge (cSB), an entropic optimal-transport sampler that factorizes layout synthesis into two coupled stochastic stages. The first stage, Region Search, transports noise toward query-aligned image zones while obeying a hard occlusion barrier that protects salient foreground content; the second stage, Appearance Shaping, sizes the glyphs through an ``ink-budget'' regularizer so that the rendered text stays legible and visually balanced. The resulting overlay behaves as an explicit attention beacon that channels the model's focus along spatial semantics. On the VMCBench suite, Timage paired with a modest 7B backbone clearly overtakes far larger proprietary systems as well as parameter-tuned baselines. The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.

Problem

• multimodal reasoning
• spatial alignment
• vision-language models
• fine-grained localization
• text-image grounding

Innovation

• Timage
• Constrained Schrödinger Bridge
• text-in-image overlay
• multimodal alignment
• input reconstruction