Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval

2026-09-14

AI Summary

该研究针对多模态检索中对称聚合方法效果不佳的问题，提出了一种基于查询条件的球面质心聚合方法SCALAR，通过学习自适应相关性权重来提高检索性能。

Abstract

Multimodal retrieval integrates video, audio, subtitles, and text; however, recent geometric aggregators, such as Gramian volumes, hyperbolic volumes, and spectral objectives, treat all modalities symmetrically. Under a unified evaluation protocol, their joint scores frequently lag behind the strongest single-modality pathway by 1.9 to 27.6 R@1. Controlled analyses attribute this outcome to uniform modality influence. This work introduces Spherical Centroid Aggregation with Learned Adaptive Relevance (SCALAR), a query-conditioned aggregator that assigns relevance-based weights to each available modality before computing a spherical centroid. SCALAR accommodates arbitrary modality subsets and is trained on masked, reduced-arity views using rank-8 LoRA adapters. Across five benchmarks, SCALAR achieves positive aggregation gain on four, reaching +4.0 R@1, while none of the evaluated prior aggregators is positive on more than one. A uniform-weight ablation reproduces the degradation observed with symmetric aggregation. With only 4.8 million trainable parameters, SCALAR attains the highest text-to-video R@1 on three and performs within seed variation of the best result on a fourth. Under test-time modality dropout, SCALAR's representation-stage score surpasses the released GRAM checkpoint at every evaluated masking rate and benchmark by 3.2 to 10.9 R@1. Finally, as modalities are removed, rerankers trained exclusively on complete modality sets increasingly converge toward their video-only pathways, diminishing these representation-level gains and underscoring a limitation of standard two-stage retrieval pipelines.

Problem

• Multimodal Retrieval
• Geometric Aggregators
• Modality Influence
• Symmetric Aggregation
• R@1

Innovation

• Spherical Centroid Aggregation
• Learned Adaptive Relevance
• Query-Conditioned Aggregator
• Rank-8 LoRA Adapters
• Modality Dropout