CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales

2026-06-20

AI Summary

Existing video captioning benchmarks struggle to comprehensively evaluate description accuracy across varying durations and scenes, as well as the consistency of entity reference over time. To address this gap, this work presents the first systematic effort to incorporate referential consistency into evaluation protocols, introducing a unified benchmark that supports both audio-visual and vision-only modalities and spans diverse video genres and temporal scales. The benchmark integrates multi-dimensional human annotations with automatic metrics under a standardized evaluation framework. Experimental results reveal a significant degradation in both caption quality and referential consistency for current models on long-form videos. Moreover, the proposed metrics exhibit strong correlation with downstream task performance, thereby validating the necessity and effectiveness of the benchmark.

Abstract

Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, few existing benchmarks can objectively and comprehensively evaluate these properties across diverse durations and scenarios, thereby hindering the advancement of video captioning models. To bridge this gap, we propose CapRiCorn-1K, a comprehensive benchmark designed to evaluate both video captioning quality and subject referential consistency across long temporal horizons and diverse video domains. To accommodate varied evaluation needs, our benchmark supports both audiovisual and visual-only settings. Extensive experiments on CapRiCorn-1K reveal that current models generally struggle to generate accurate and comprehensive captions while maintaining consistent subject references. Moreover, as video duration increases, both the overall caption quality and subject referential consistency decline. Notably, our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions, further validating their effectiveness. The project is available at https://github.com/xlchen0205/CapRiCorn-1K .

Problem

• video captioning
• subject referential consistency
• temporal scales
• benchmark evaluation

Innovation

• video captioning
• referential consistency
• temporal scale
• comprehensive benchmark
• subject consistency