From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

2026-06-13

AI Summary

This work addresses the challenge of fine-grained hand-object interaction recognition in egocentric videos by proposing a tuning-free symbolic reasoning paradigm that circumvents the limitations of general-purpose vision-language models (VLMs) when directly processing raw pixels. The approach transforms input videos into temporal action graphs: first generating short-window natural language narratives via multi-stage prompting, then structuring these into an open-vocabulary, graph-based representation amenable to symbolic-level in-context learning and reasoning. Experiments demonstrate that the method significantly outperforms existing zero-shot frame-level and graph-level baselines on EGTEA and Epic-Kitchens-100, achieving performance on par with pixel-level models. Furthermore, its effectiveness is validated across 11 mainstream VLMs, revealing that VLMs are better suited as symbolic reasoners than as direct visual observers.

Abstract

Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference alike. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.

Problem

• egocentric action recognition
• temporal reasoning
• hand-object interaction
• vision-language models
• fine-grained action understanding

Innovation

• Temporal Action Graphs
• In-Context Learning
• Vision-Language Models
• Egocentric Action Recognition
• Symbolic Reasoning