Unifying Specialized Visual Encoders for Video Language Models

2025-01-02

AI Summary

Current video-language models (VideoLLMs) typically rely on a single visual encoder, limiting their capacity to capture fine-grained spatiotemporal dynamics and diverse visual knowledge. To address this, we propose MERV—a novel framework that enables collaborative inference among multiple frozen visual encoders for the first time. MERV aligns cross-modal spatiotemporal features to fuse heterogeneous encoder representations into a unified, robust video semantic representation. Crucially, it requires no encoder fine-tuning and introduces zero additional parameters, while supporting parallel visual processing. On standard video understanding benchmarks, MERV achieves up to a 3.7% absolute accuracy gain over Video-LLaVA; in zero-shot perception tasks, it outperforms SeViLA by 2.2%; and it demonstrates significantly improved inference efficiency, as validated by higher scores on the Video-ChatGPT evaluation suite.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The recent advent of Large Language Models (LLMs) has ushered sophisticated reasoning capabilities into the realm of video through Video Large Language Models (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder methods while parallelizing the visual processing. Finally, we provide qualitative evidence that MERV successfully captures domain knowledge from each of its encoders. Our results offer promising directions in utilizing multiple vision encoders for comprehensive video understanding.

Problem

• Video Language Model
• Visual Encoder
• Multimodal Understanding

Innovation

• MERV
• Multi-Visual Encoders
• Enhanced Video Understanding