Vision-Language Grounding as Bidirectional Concept Correspondence

2026-08-07

AI Summary

This work addresses a key limitation in existing vision–language alignment methods, which typically assume known textual spans and support only unidirectional grounding, thereby overlooking the core challenge of identifying visually grounded phrases in text and establishing their correspondence with image regions. To overcome this, the paper introduces ConCor-1, the first bidirectional concept correspondence framework that formulates alignment as a fine-grained, bidirectional matching problem between images and text without requiring predefined textual segments. By leveraging learnable bridge tokens, ConCor-1 jointly predicts text masks, image masks, and cross-modal correspondence scores in a unified, end-to-end manner, enabling simultaneous segmentation and alignment. The method achieves a 48% F1 improvement on datasets with long textual descriptions and demonstrates a 29% performance gain on zero-shot LVIS tasks, substantially outperforming current baselines.

Abstract

Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.

Problem

• vision-language grounding
• bidirectional concept correspondence
• visually referential text
• cross-modal alignment
• grounded communication

Innovation

• bidirectional concept correspondence
• vision-language grounding
• bridge tokens
• open-vocabulary detection
• cross-modal alignment