Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

2026-03-03

AI Summary

This work addresses the challenge of real-time video commentary, which requires joint optimization of *what* to say and *when* to say it—a temporal alignment often overlooked by existing approaches. The authors propose a fine-tuning-free multimodal large language model framework that leverages contextual prompting to generate semantically relevant and temporally appropriate commentary. The key innovation is a dynamic interval decoding strategy that adaptively schedules the generation of each subsequent sentence based on the predicted duration of the preceding one, enabling pause-aware real-time output. Evaluated on bilingual (Japanese and English) datasets from racing and fighting games, the method significantly improves temporal alignment between generated commentary and human speech patterns. The study also contributes an open-sourced multilingual benchmark dataset and codebase to support future research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.

Problem

• real-time commentary generation
• multimodal LLMs
• timing alignment
• pause-aware decoding
• video-to-text generation

Innovation

• real-time commentary generation
• multimodal LLMs
• pause-aware decoding
• dynamic interval decoding
• prompting-based approach