Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

2026-09-14

AI Summary

该研究通过引入Hypergraph-Regularized Gramian Volumes方法，在多模态检索中利用语义关系优化候选样本的嵌入表示，从而提高文本与视频、音频及字幕匹配的准确性。

Abstract

Volume-based multimodal retrieval jointly scores a text query with a candidate's video, audio, and subtitle embeddings. While this approach captures higher-order within-candidate alignment, the score remains candidate-local, and semantically related training samples primarily serve as contrastive negatives. This work introduces Hypergraph-Regularized Gramian Volumes (HyVol), a training-time module that incorporates these semantic relations prior to evaluating the original volume loss. Document hyperedges connect the observed modalities of each candidate, whereas semantic hyperedges link candidates whose detached captions are mutual top-k neighbors. A shallow gated hyper-graph network applies residual corrections to the modality embeddings. Presence masks exclude unavailable streams from message passing, and identity padding preserves the determinant of the observed Gram submatrix without feature imputation. As refinement operates on embeddings rather than scores, the same construction applies to both Gram and HyperGram. We remove the hypergraph after training, leaving the backbone-only architecture, original scoring function, and retrieval cost unchanged. We train both backbones on a 150K-clip subset of VAST-27M and evaluate zero-shot performance on six benchmarks. Under the paired protocol, HyVol improves R@1 across all five retrieval benchmarks, with video-to-text gains reaching +8.3 on MSR-VTT and +7.6 on VATEX. Under missing-modality masking, the V2T margin remains positive in all experimental settings, although the T2V margin becomes slightly negative in four.

Problem

• Multimodal Retrieval
• Semantic Relations
• Volume-based
• Contrastive Negatives
• Candidate-local

Innovation

• Hypergraph-Regularized
• Gramian Volumes
• Multimodal Retrieval
• Semantic Relations
• Residual Corrections