Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos

2026-04-19

AI Summary

This work addresses the problem of generating intermediate states during object state transformations from an egocentric viewpoint—a task newly defined as Egocentric Intermediate Visual State Transformation (EIVST)—by introducing the EgoIn framework. EgoIn leverages a fine-tuned vision-language model, TransitionVLM, for multi-step action reasoning and incorporates a Transition Conditioning module to generate intermediate frames conditioned on natural language instructions. To preserve object appearance consistency throughout the transformation sequence, the framework further introduces Object-aware Auxiliary Supervision. Experimental results on datasets capturing human-object and robot-object interactions demonstrate that EgoIn substantially outperforms existing approaches, producing sequences that exhibit superior semantic plausibility and visual coherence.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: State Estimation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Understanding physical transformation processes is crucial for both human cognition and artificial intelligence systems, particularly from an egocentric perspective, which serves as a key bridge between humans and machines in action modeling. We define this modeling process as Egocentric Instructed Visual State Transition (EIVST), which involves generating intermediate frames that depict object transformations between initial and target states under a brief action instruction. EIVST poses two challenges for current generative models: (1) understanding the visual scenes of the initial and target states and reasoning about transformation steps from an egocentric view, and (2) generating a consistent intermediate transition that follows the given instruction while preserving object appearance across the two visual states. To address these challenges, we propose the EgoIn framework. It first infers the multi-step transition process between two given states using TransitionVLM, fine-tuned on our curated dataset to better adapt to this task and reduce hallucinated information. It then generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module. Additionally, we introduce Object-aware Auxiliary Supervision to preserve consistent object appearance throughout the transition. Extensive experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences.

Problem

• Egocentric video
• Object state transition
• Intermediate frame generation
• Visual consistency
• Action instruction

Innovation

• Egocentric Video Generation
• Visual State Transition
• TransitionVLM
• Object-aware Supervision
• Instruction-conditioned Generation