Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

2026-07-13

AI Summary

This study addresses the concern that high benchmark scores of current video large language models (Video-LLMs) on long-video character tracking tasks may stem from reliance on coarse-grained cues—such as gender—rather than genuine understanding of character identity. To investigate this, the authors propose a nine-condition diagnostic protocol employing controlled experiments including character name substitution, open-ended questioning, and enhancements via subtitles and frame counts. Systematic evaluation of multiple open-source Video-LLMs alongside Gemini 2.5 Flash reveals that models adjust their answers in only 4–31% of cases following name changes, struggle to distinguish between same-gender characters, and exhibit substantially reduced accuracy in open-ended settings, with no model achieving full correctness. This work is the first to expose the superficial mechanisms underlying Video-LLM character tracking and releases a diagnostic toolkit to audit the authenticity of benchmark performance.

Abstract

Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini~2.5~Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.

Problem

• Video-LLM
• character tracking
• long-form video
• visual reasoning
• benchmark evaluation

Innovation

• Video-LLM
• character tracking
• diagnostic protocol
• gender bias
• benchmark evaluation