SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos

2025-04-10

AI Summary

Existing video scene graph generation (VidSGG) methods for first-person kitchen videos suffer from heavy reliance on large-scale training data and exhibit temporal inconsistency in object identity tracking across frames—particularly when leveraging multimodal large language models (e.g., Gemini). To address this, we propose the first training-free, zero-shot VidSGG framework. Our method uniquely integrates SAM2-based temporal mask propagation with Gemini’s multimodal semantic parsing, and introduces a novel graph-mask matching algorithm to achieve fine-grained object binding and semantic alignment—preserving precise bounding-box localization while substantially improving cross-frame object identity consistency. Evaluated on EPIC-KITCHENS and EPIC-KITCHENS-100, our approach achieves an 8.33% absolute improvement in mean recall over Gemini, demonstrating for the first time the feasibility of generating high-quality, temporally consistent scene graphs in dynamic, complex environments under a zero-shot paradigm.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video Scene Graph Generation (VidSGG) is an important topic in understanding dynamic kitchen environments. Current models for VidSGG require extensive training to produce scene graphs. Recently, Vision Language Models (VLM) and Vision Foundation Models (VFM) have demonstrated impressive zero-shot capabilities in a variety of tasks. However, VLMs like Gemini struggle with the dynamics for VidSGG, failing to maintain stable object identities across frames. To overcome this limitation, we propose SAMJAM, a zero-shot pipeline that combines SAM2's temporal tracking with Gemini's semantic understanding. SAM2 also improves upon Gemini's object grounding by producing more accurate bounding boxes. In our method, we first prompt Gemini to generate a frame-level scene graph. Then, we employ a matching algorithm to map each object in the scene graph with a SAM2-generated or SAM2-propagated mask, producing a temporally-consistent scene graph in dynamic environments. Finally, we repeat this process again in each of the following frames. We empirically demonstrate that SAMJAM outperforms Gemini by 8.33% in mean recall on the EPIC-KITCHENS and EPIC-KITCHENS-100 datasets.

Problem

• Zero-shot video scene graph generation in kitchens
• Maintaining object identities across video frames
• Improving object grounding and temporal consistency

Innovation

• Combines SAM2 tracking with Gemini semantics
• Uses matching algorithm for consistent object identities
• Improves object grounding with accurate bounding boxes