DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval

2026-06-17

AI Summary

Existing video–language retrieval methods struggle to capture fine-grained temporal dynamics and complex linguistic semantics, limiting cross-modal alignment performance. To address this, this work proposes the DREAM framework, which enhances multimodal representation through hierarchical visual encoding and hybrid language modeling that integrates masked and permuted language modeling objectives. A cascaded grouped attention mechanism is introduced to enable coarse-to-fine spatiotemporal modeling, complemented by multi-stage token interactions to refine alignment accuracy. The proposed approach achieves state-of-the-art results on standard benchmarks, yielding R@1 scores of 49.4%, 49.7%, and 27.3% on the MSRVTT, MSVD, and LSMDC datasets, respectively, significantly outperforming existing methods.

Abstract

In today's media-driven world, the exponential growth of video content across domains such as surveillance, education, and entertainment has made retrieving semantically relevant videos via natural language queries increasingly critical. Early video retrieval systems relied on handcrafted features or shallow cross-modal mappings, limiting their ability to capture complex semantics and temporal dynamics. While large-scale vision-language models have improved cross-modal alignment, challenges remain in modeling fine-grained temporal dependencies and nuanced linguistic structures. In this paper, we introduce DREAM: Dual-path Representation Enhancement and Alignment Model, a novel multimodal framework that addresses these limitations through enhanced visual and textual encoding. DREAM incorporates a hybrid language modeling strategy that combines masked and permuted language modeling objectives to capture both local and global linguistic semantics. On the visual side, we design a hierarchical vision encoder with cascaded group attention, which integrates spatial and temporal information through multi-stage token interaction and coarse-to-fine attention refinement. We validate DREAM through comprehensive evaluations on the widely-used MSRVTT, MSVD and LSMDC benchmark datasets, where it achieves new state-of-the-art R1 scores of 49.4%, 49.7% and 27.3%, respectively. Qualitative analyses further show the model's ability to maintain coherent attention across frames and align complex queries with dynamic video content. These findings underscore the effectiveness of hierarchical attention and dual-objective textual modeling in enabling robust, context-aware video retrieval, and pave the way for future research in advancing cross-modal representation learning.

Problem

• cross-modal retrieval
• video retrieval
• temporal dependencies
• linguistic semantics
• vision-language models

Innovation

• dual-objective language modeling
• hierarchical vision encoder
• cascaded group attention
• cross-modal retrieval
• temporal dynamics modeling