OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation

2026-05-26

AI Summary

Existing audio-visual-text (AVT) encoders fail to effectively leverage tri-modal fused embeddings during training, limiting cross-modal retrieval performance. This work proposes a “Fusion-as-Teacher” distillation mechanism that uses frozen fused embeddings as supervision signals for unimodal representations, coupled with a Tuple-InfoNCE loss to directly optimize the fused representation. This approach enables efficient zero-shot retrieval across any pair of modalities. Evaluated on six public benchmarks, the method surpasses Gemini Embedding 2 by 13.3–18.0 R@1 and achieves an AVG-all score of 34.84 on the newly introduced OmniRetriever-Bench, outperforming the best open-source baseline by 8.03, thereby demonstrating its effectiveness and state-of-the-art performance in open-domain multimodal retrieval.

Abstract

Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.

Problem

• multimodal retrieval
• audio-video-text embedding
• cross-modal representation
• joint embedding
• zero-shot retrieval

Innovation

• fusion-as-teacher distillation
• multimodal embedding
• audio-video-text retrieval
• Tuple-InfoNCE
• zero-shot retrieval