Towards Efficient and Robust Moment Retrieval System: A Unified Framework for Multi-Granularity Models and Temporal Reranking

2025-04-11

AI Summary

Long-video understanding faces key bottlenecks in interactive retrieval: limited model diversity, storage redundancy, inaccurate temporal localization, and context-agnostic re-ranking. To address these, we propose an efficient and robust moment retrieval framework for long videos. Our approach features four core innovations: (1) multi-granularity model ensemble search, integrating CLIP and BEIT-3; (2) representative keyframe sampling and deduplicated storage based on TransNetV2; (3) dual-query-driven fine-grained temporal localization; and (4) neighbor-frame-aware contextual re-ranking. Extensive experiments demonstrate significant improvements in retrieval accuracy, response latency, and result interpretability across both known-item retrieval and video question answering tasks. The framework effectively supports real-world interactive long-video retrieval with enhanced efficiency and robustness.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Long-form video understanding presents significant challenges for interactive retrieval systems, as conventional methods struggle to process extensive video content efficiently. Existing approaches often rely on single models, inefficient storage, unstable temporal search, and context-agnostic reranking, limiting their effectiveness. This paper presents a novel framework to enhance interactive video retrieval through four key innovations: (1) an ensemble search strategy that integrates coarse-grained (CLIP) and fine-grained (BEIT3) models to improve retrieval accuracy, (2) a storage optimization technique that reduces redundancy by selecting representative keyframes via TransNetV2 and deduplication, (3) a temporal search mechanism that localizes video segments using dual queries for start and end points, and (4) a temporal reranking approach that leverages neighboring frame context to stabilize rankings. Evaluated on known-item search and question-answering tasks, our framework demonstrates substantial improvements in retrieval precision, efficiency, and user interpretability, offering a robust solution for real-world interactive video retrieval applications.

Problem

• Enhance interactive video retrieval accuracy using multi-granularity models
• Optimize storage by reducing redundancy in video keyframes
• Improve temporal search stability with context-aware reranking

Innovation

• Ensemble search with CLIP and BEIT3 models
• Storage optimization via TransNetV2 deduplication
• Temporal search with dual query localization