Wolf: Dense Video Captioning with a World Summarization Framework

2024-07-26

AI Summary

To address the need for fine-grained, temporally aligned video captioning in autonomous driving, general-purpose scenes, and robotics, this paper introduces Wolf, a novel “world summarization” framework. Methodologically, Wolf pioneers the world summarization paradigm, employing a Mixture-of-Experts architecture to synergistically integrate heterogeneous vision-language models (VLMs) for both image and video understanding, enabling robust cross-modal temporal modeling. It further proposes CapScore—a large language model–driven evaluation metric—that jointly quantifies semantic similarity and caption quality for the first time. Additionally, we construct the first high-quality, human-annotated video captioning benchmark spanning all three application domains, accompanied by a public leaderboard. Experiments demonstrate that Wolf achieves state-of-the-art performance across multiple benchmarks. On challenging driving videos, Wolf attains a 55.6% improvement in CapScore quality and a 77.4% gain in semantic similarity over GPT-4V.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment. Webpage: https://wolfv0.github.io/.

Problem

• Develops an automated video captioning framework using Vision Language Models.
• Introduces CapScore to evaluate caption quality and similarity.
• Establishes a benchmark and leaderboard for video captioning advancements.

Innovation

• Mixture-of-experts approach using Vision Language Models
• CapScore metric for evaluating caption quality
• Four human-annotated datasets for comprehensive comparisons