Language-driven Description Generation and Common Sense Reasoning for Video Action Recognition

2025-06-20

AI Summary

To address degraded action recognition performance in monocular videos caused by severe occlusion and cluttered scenes, this work introduces large language model (LLM)-encoded commonsense priors explicitly into the action recognition pipeline for the first time. Methodologically, we propose a three-stage framework: (1) video context summarization; (2) prompt-enhanced, commonsense-driven scene text description modeling; and (3) vision–language cross-modal fusion with sequence-level commonsense reasoning. This design enables explicit modeling of implicit contextual cues—including human–object interactions, object functionality, and activity logic—thereby significantly improving occlusion robustness. Experiments on Action Genome and Charades demonstrate that our method achieves average accuracy gains of 5.2%–8.7% on occlusion-heavy subsets, validating the effectiveness of integrating structured commonsense knowledge for complex action understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent video action recognition methods have shown excellent performance by adapting large-scale pre-trained language-image models to the video domain. However, language models contain rich common sense priors - the scene contexts that humans use to constitute an understanding of objects, human-object interactions, and activities - that have not been fully exploited. In this paper, we introduce a framework incorporating language-driven common sense priors to identify cluttered video action sequences from monocular views that are often heavily occluded. We propose: (1) A video context summary component that generates candidate objects, activities, and the interactions between objects and activities; (2) A description generation module that describes the current scene given the context and infers subsequent activities, through auxiliary prompts and common sense reasoning; (3) A multi-modal activity recognition head that combines visual and textual cues to recognize video actions. We demonstrate the effectiveness of our approach on the challenging Action Genome and Charades datasets.

Problem

• Exploiting language models' common sense priors for video action recognition
• Generating descriptions and reasoning about cluttered, occluded video sequences
• Combining visual and textual cues to improve activity recognition accuracy

Innovation

• Video context summary for objects and interactions
• Description generation with common sense reasoning
• Multi-modal recognition combining visual and textual cues