Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding

2026-04-13

AI Summary

This study investigates the impact and limitations of "chain-of-thought" reasoning—i.e., internal inference traces—on output quality in video scene understanding using vision-language models. Leveraging the Gemini 2.5 Flash model series, the authors conduct a systematic evaluation on 100 hours of video data to analyze the effects of varying reasoning lengths, content transformation, and attentional focus. They introduce three novel metrics—Contentfulness, Thought-Final Coverage, and Dominant Entity Analysis—and employ GPT-5 as an independent evaluator for both quantitative and qualitative assessment. The findings reveal that performance gains from extended reasoning saturate rapidly, with Flash Lite achieving the best trade-off between output quality and token efficiency. Moreover, overly constrained reasoning budgets tend to induce “compression hallucinations,” wherein the model generates outputs without sufficient intermediate reasoning.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We benchmark how internal reasoning traces, which we call thought streams, affect video scene understanding in vision-language models. Using four configurations of Google's Gemini 2.5 Flash and Flash Lite across scenes extracted from 100 hours of video, we ask three questions: does more thinking lead to better outputs, where do the gains stop, and what do these models actually think about? We introduce three evaluation metrics. Contentfulness measures how much of the thought stream is useful scene content versus meta-commentary. Thought-Final Coverage measures how faithfully the thought stream translates into the final output. Dominant Entity Analysis identifies which subjects, actions, and settings the model focuses on. GPT-5 serves as an independent judge. We find that quality gains from additional thinking plateau quickly, with most improvement occurring in the first few hundred tokens. Flash Lite offers the best balance between quality and token usage. Tight reasoning budgets cause the model to add content in the final output that it never reasoned about, a form of compression-step hallucination. Despite being different model tiers, Flash and Flash Lite produce similar thought streams, though they differ in style: Flash discusses its reasoning process, while Lite focuses on describing the scene.

Problem

• thought streams
• video scene understanding
• vision-language models
• reasoning evaluation
• model hallucination

Innovation

• thought streams
• video scene understanding
• reasoning evaluation metrics
• compression-step hallucination
• vision-language models