Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Models

2026-06-12

AI Summary

This work addresses the structural output repetition and sequential anomalies that commonly arise when fine-tuning vision-language models to generate dense coordinate lists, which severely compromise localization controllability. The study identifies these issues as stemming from a separable and quantifiable "structural binding interference surface." To mitigate this, the authors propose high-capacity LoRA fine-tuning applied to query, key, value, and output projections, complemented by structural-axis probing analysis and an object-level repetition-suppression intervention mechanism. Evaluated on Gemma-4-12B, the approach boosts F1@0.3 from 0.007 to 0.490 while eliminating repetitions entirely; similarly strong results are achieved on Qwen3-VL-8B with F1@0.3=0.318 and zero repetition. Cross-model reproducibility is further validated on the COCO 2017 benchmark.

Abstract

Fine-tuning vision-language models to emit dense coordinate lists improves visual grounding but also changes how models serialize, repeat, and terminate structured outputs. We study this behavior as a generation and control surface. In Gemma 4 12B, high-capacity q/k/v/o LoRA raises class-aware F1@0.3 from 0.007 to 0.448 while inducing repeated-tail pressure (duplicate rate 0.080, max repeat 23). A q/v rank sweep keeps max repeat at 21-22 across ranks 4-64, showing capacity persistence. The target signal is separable: object-level repeat-stop removes exact repeated records (duplicate rate 0.000, max repeat 1) while preserving F1 (0.494 to 0.490) and stricter F1@0.5 (0.381 to 0.385). Structure-axis probes localize the effect to bbox-coordinate object lists; dense non-bbox and spatial/count JSON remain repeat-clean, including under high-capacity adapters. Qwen3-VL-8B reproduces a clean controlled endpoint (F1@0.3 0.318, duplicate rate 0.000), and COCO 2017 reproduces acquisition plus duplicate pressure. Dense coordinate-list adaptation therefore creates a structure-bound, cross-family interference surface that can be measured and controlled.

Problem

• dense coordinate-list
• visual grounding
• structured output repetition
• vision-language models
• interference surface

Innovation

• dense coordinate-list fine-tuning
• controllable interference surface
• structured output repetition
• visual grounding
• LoRA adaptation