LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension

2025-11-14

AI Summary

Existing weakly supervised referring expression comprehension (WREC) methods rely on a restrictive one-to-one mapping assumption, failing to handle real-world scenarios involving zero-reference or one-expression-to-multiple-targets cases. This paper introduces generalized weakly supervised referring expression comprehension (WGREC), a new task explicitly addressing two core challenges: ambiguous supervision signals and semantic representation collapse. To tackle them, we propose a two-stage decoupled framework and present LIHE—a novel hyperbolic-Euclidean hybrid similarity modeling method. LIHE employs a language-instance splitting mechanism for expression decomposition and integrates the HEMix module, which ensures fine-grained alignment in Euclidean space while capturing hierarchical semantic structures in hyperbolic space. We establish the first WGREC weakly supervised baselines on gRefCOCO and Ref-ZOM. Empirically, HEMix achieves a 2.5% improvement in IoU@0.5 over state-of-the-art methods on standard REC benchmarks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Existing Weakly-Supervised Referring Expression Comprehension (WREC) methods, while effective, are fundamentally limited by a one-to-one mapping assumption, hindering their ability to handle expressions corresponding to zero or multiple targets in realistic scenarios. To bridge this gap, we introduce the Weakly-Supervised Generalized Referring Expression Comprehension task (WGREC), a more practical paradigm that handles expressions with variable numbers of referents. However, extending WREC to WGREC presents two fundamental challenges: supervisory signal ambiguity, where weak image-level supervision is insufficient for training a model to infer the correct number and identity of referents, and semantic representation collapse, where standard Euclidean similarity forces hierarchically-related concepts into non-discriminative clusters, blurring categorical boundaries. To tackle these challenges, we propose a novel WGREC framework named Linguistic Instance-Split Hyperbolic-Euclidean (LIHE), which operates in two stages. The first stage, Referential Decoupling, predicts the number of target objects and decomposes the complex expression into simpler sub-expressions. The second stage, Referent Grounding, then localizes these sub-expressions using HEMix, our innovative hybrid similarity module that synergistically combines the precise alignment capabilities of Euclidean proximity with the hierarchical modeling strengths of hyperbolic geometry. This hybrid approach effectively prevents semantic collapse while preserving fine-grained distinctions between related concepts. Extensive experiments demonstrate LIHE establishes the first effective weakly supervised WGREC baseline on gRefCOCO and Ref-ZOM, while HEMix achieves consistent improvements on standard REC benchmarks, improving IoU@0.5 by up to 2.5%. The code is available at https://anonymous.4open.science/r/LIHE.

Problem

• Extends weakly-supervised referring expression comprehension to handle variable numbers of referents
• Addresses supervisory signal ambiguity in identifying correct number and identity of referents
• Solves semantic representation collapse where hierarchical concepts become non-discriminative

Innovation

• Linguistic instance-split framework for generalized referring expression
• Hybrid Euclidean-hyperbolic similarity module HEMix
• Two-stage referential decoupling and grounding approach