🤖 AI Summary
To address three key challenges in video moment retrieval—poor cross-modal noise robustness, weak modeling of event temporal coherence, and heavy reliance on manual modality selection—this paper proposes an end-to-end interactive multimodal moment retrieval framework. Methodologically: (1) a cascaded dual-embedding re-ranking architecture enhances cross-modal alignment; (2) a temporal-aware exponentially decaying scoring mechanism explicitly models event continuity while suppressing implausible time intervals; (3) GPT-4o enables automatic query decomposition and adaptive score fusion, eliminating manual modality specification. The system integrates BEIT-3, SigLIP, BLIP-2, and ASR/OCR features, with beam search for temporally constrained optimization. Experiments demonstrate substantial improvements in recall-precision trade-off under ambiguous queries, generation of highly coherent event sequences, and enhanced practicality of video retrieval systems.
📝 Abstract
The exponential growth of video content has created an urgent need for efficient multimodal moment retrieval systems. However, existing approaches face three critical challenges: (1) fixed-weight fusion strategies fail across cross modal noise and ambiguous queries, (2) temporal modeling struggles to capture coherent event sequences while penalizing unrealistic gaps, and (3) systems require manual modality selection, reducing usability. We propose a unified multimodal moment retrieval system with three key innovations. First, a cascaded dual-embedding pipeline combines BEIT-3 and SigLIP for broad retrieval, refined by BLIP-2 based reranking to balance recall and precision. Second, a temporal-aware scoring mechanism applies exponential decay penalties to large temporal gaps via beam search, constructing coherent event sequences rather than isolated frames. Third, Agent-guided query decomposition (GPT-4o) automatically interprets ambiguous queries, decomposes them into modality specific sub-queries (visual/OCR/ASR), and performs adaptive score fusion eliminating manual modality selection. Qualitative analysis demonstrates that our system effectively handles ambiguous queries, retrieves temporally coherent sequences, and dynamically adapts fusion strategies, advancing interactive moment search capabilities.