Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension

2026-07-05

AI Summary

This work addresses the limitation of existing visual representations in weakly supervised referring expression comprehension, where the lack of structured organization hinders alignment with the compositional nature of language. To overcome this, the paper proposes the first framework that explicitly models structured visual compositional representations by jointly learning unary object and binary relation embeddings. A unified compositional alignment mechanism is introduced to enable fine-grained vision–language matching under weak supervision. The method achieves state-of-the-art performance across the RefCOCO, RefCOCO+, and RefCOCOg benchmarks, demonstrating the effectiveness and advantage of structured visual representations coupled with compositional alignment in weakly supervised settings.

Abstract

Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.

Problem

• Weakly Supervised Referring Expression Comprehension
• Structured Visual Representation
• Visual-Textual Alignment
• Compositional Representation
• Relational Embedding

Innovation

• structured visual representation
• compositional alignment
• weakly supervised referring expression comprehension
• relational embedding
• visual-textual matching