Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

2026-08-29

AI Summary

为解决CLIP类模型无法编码部分-整体等关系结构的问题，本文提出Hyper3-CLIP，结合超球面几何与层次条件查询进行视觉-语言训练。

Abstract

CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.

Problem

• Vision-Language Models
• Relational Structure
• Hyperbolic Embeddings

Innovation

• Hierarchy-Conditioned
• Hyperbolic VLMs
• Query-Conditioned Pooling
• Contrastive Learning