Addressing the ID-Matching Challenge in Long Video Captioning

2025-10-08

AI Summary

In long-video captioning, achieving cross-frame consistent identification of the same person (ID matching) remains a critical challenge. This paper proposes RICE (Recognition-based Identity Consistency Enhancement), the first method to systematically uncover and activate the intrinsic ID-matching capability of large vision-language models (LVLMs)—e.g., GPT-4o—by abandoning conventional pairwise matching paradigms. Instead, RICE enhances identity-aware video understanding and caption generation end-to-end via image-informed context enrichment and individualized descriptive expansion. We introduce the first dedicated ID-matching benchmark for long-video captioning. Experiments on GPT-4o demonstrate substantial improvements: ID-matching accuracy reaches 90% (+40 percentage points) and recall attains 80% (+65 percentage points) over prior baselines. These gains enable robust, multi-character, long-horizon tracking and generate temporally coherent, identity-consistent captions.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Generating captions for long and complex videos is both critical and challenging, with significant implications for the growing fields of text-to-video generation and multi-modal understanding. One key challenge in long video captioning is accurately recognizing the same individuals who appear in different frames, which we refer to as the ID-Matching problem. Few prior works have focused on this important issue. Those that have, usually suffer from limited generalization and depend on point-wise matching, which limits their overall effectiveness. In this paper, unlike previous approaches, we build upon LVLMs to leverage their powerful priors. We aim to unlock the inherent ID-Matching capabilities within LVLMs themselves to enhance the ID-Matching performance of captions. Specifically, we first introduce a new benchmark for assessing the ID-Matching capabilities of video captions. Using this benchmark, we investigate LVLMs containing GPT-4o, revealing key insights that the performance of ID-Matching can be improved through two methods: 1) enhancing the usage of image information and 2) increasing the quantity of information of individual descriptions. Based on these insights, we propose a novel video captioning method called Recognizing Identities for Captioning Effectively (RICE). Extensive experiments including assessments of caption quality and ID-Matching performance, demonstrate the superiority of our approach. Notably, when implemented on GPT-4o, our RICE improves the precision of ID-Matching from 50% to 90% and improves the recall of ID-Matching from 15% to 80% compared to baseline. RICE makes it possible to continuously track different individuals in the captions of long videos.

Problem

• Addressing ID-Matching challenge in long video captioning
• Improving individual recognition across different video frames
• Enhancing identity tracking accuracy in complex video descriptions

Innovation

• Leveraging LVLMs for inherent ID-Matching capabilities
• Enhancing image information usage for better recognition
• Increasing individual description quantity in captions