Long Video Understanding with Learnable Retrieval in Video-Language Models

2023-12-08

AI Summary

To address the computational overhead, loss of fine-grained visual details, and interference from irrelevant noise caused by excessive visual tokens in long-video understanding, this paper proposes a question-driven, learnable video snippet retrieval framework. Methodologically, we design a lightweight, end-to-end trainable MLP-based retrieval module coupled with a soft matching loss to dynamically select the top-K video snippets most relevant to the given question; only their visual tokens are fed into the large language model (LLM). We further integrate video chunk encoding with vision-language joint modeling to enable zero-shot cross-dataset generalization. Compared to fixed-sampling or global-aggregation paradigms, our approach significantly improves fine-grained reasoning accuracy, outperforms existing baselines on multiple zero-shot video question answering benchmarks, and simultaneously reduces GPU memory consumption and computational cost.

Abstract

The remarkable natural language understanding, reasoning, and generation capabilities of large language models (LLMs) have made them attractive for application to video understanding, utilizing video tokens as contextual input. However, employing LLMs for long video understanding presents significant challenges. The extensive number of video tokens leads to considerable computational costs for LLMs while using aggregated tokens results in loss of vision details. Moreover, the presence of abundant question-irrelevant tokens introduces noise to the video reasoning process. To address these issues, we introduce a simple yet effective learnable retrieval-based video-language model (R-VLM) for efficient long video understanding. Specifically, given a question (query) and a long video, our model identifies and selects the most relevant K video chunks and uses their associated visual tokens to serve as context for the LLM inference. This effectively reduces the number of video tokens, eliminates noise interference, and enhances system performance. We achieve this by incorporating a learnable lightweight MLP block to facilitate the efficient retrieval of question-relevant chunks, through the end-to-end training of our video-language model with a proposed soft matching loss. Our experimental results on multiple zero-shot video question answering datasets validate the effectiveness of our framework for comprehending long videos.

Problem

• Efficient long video understanding
• Reducing computational costs
• Eliminating irrelevant video tokens

Innovation

• Learnable retrieval-based video-language model
• Reduces video tokens and eliminates noise
• End-to-end training with soft matching loss