Scaling Representation Diversity: Modulated Attention and Reconstructive Regularization for Visual Grounding

2026-08-12

AI Summary

This work addresses the limited generalization and representation degradation in visual referring expression comprehension caused by dataset-specific fine-tuning. To this end, the authors propose a data–model co-design framework that, for the first time, leverages representation diversity as a guiding principle. The approach introduces a modulated Attention-based Contrastive Head (mACH) and a text-guided JEPA auxiliary stream, alongside the construction of Objects365-Caption—a large-scale captioned dataset—to enhance model generality. By incorporating reconstruction regularization and a complementary gradient subspace mechanism, the framework effectively preserves representation diversity across tasks. The resulting single-model architecture achieves state-of-the-art performance on multiple referring expression comprehension benchmarks and demonstrates significantly improved cross-dataset generalization without requiring task-specific adaptation.

Abstract

Referring Expression Comprehension (REC) is commonly studied under dataset-specific fine-tuning, resulting in specialist models with limited cross-dataset generalization. In this work, we revisit REC from the perspective of unified open-vocabulary grounding and identify representation degeneration as a key obstacle to scaling a single generalist model. To preserve representation diversity, we propose a holistic data-model co-design framework. Architecturally, we introduce the Modulated Attention-Contrastive Head (mACH) for efficient token-level vision-language alignment and a text-conditioned JEPA auxiliary stream that provides complementary gradient support to preserve alignment-active representations without inference overhead. On the data side, we introduce Objects365-Caption, enriching Objects365 with context-aware referring expressions for large-scale language supervision. We further provide a theoretical analysis showing that complementary gradient subspaces preserve alignment capacity and thereby scale representation diversity. Extensive experiments demonstrate that our single-checkpoint framework achieves highly competitive performance on standard REC benchmarks while exhibiting strong generalization across heterogeneous grounding datasets without benchmark-specific adaptation.

Problem

• Referring Expression Comprehension
• representation degeneration
• cross-dataset generalization
• visual grounding
• open-vocabulary grounding

Innovation

• Modulated Attention
• Reconstructive Regularization
• Representation Diversity
• Visual Grounding
• Open-vocabulary REC