Task Breakpoint Generation using Origin-Centric Graph in Virtual Reality Recordings for Adaptive Playback

2026-03-08

AI Summary

This work addresses the limitations of existing task segmentation methods, which rely on manual annotations or are confined to 2D video, and thus struggle to support adaptive replay in 3D virtual reality (VR) environments. To overcome this, the study introduces Origin-Centered Graphs (OCGs) into VR task segmentation for the first time, integrating them with Structured Spatio-Temporal Scene Graphs (STSGs) to automatically track state changes of core objects and the formation of new assemblies in procedural tasks. This approach enables annotation-free generation of task breakpoints that align closely with user perception. Furthermore, the method dynamically adjusts replay pacing based on user proficiency, demonstrating high accuracy and effectiveness in enabling adaptive VR replay without human labeling.

Technology Category

• Computer Vision: Segmentation
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We propose a method for generating task breakpoints based on an Origin-Centric Graph (OCG) to segment goal-oriented activity recordings into task units for adaptive playback in Virtual Reality (VR) environments. With the development of Augmented Reality (AR)/VR head-mounted displays (HMDs), research on adaptive tutorials and authoring tools has become active, but existing task segmentation methods mainly rely on manual annotation or are restricted to 2D video which limits their applicability to 3D VR contexts. In our approach, assembly scenarios with clearly defined task boundaries are recorded using a structured spatio-temporal scene graph (STSG), and the OCG is employed to track changes in the central object and the formation of new groups, thereby generating task breakpoints automatically. A user study collected user-perceived task breakpoints to establish ground truth (GT), and comparison with the algorithm-detected breakpoints demonstrated high agreement and confirmed accuracy in supporting adaptive playback. The proposed task segmentation method provides a foundation for dynamically adjusting VR playback according to user proficiency and progress, with potential for extension into automatic timeline segmentation systems for diverse VR recordings.

Problem

• task segmentation
• virtual reality
• adaptive playback
• 3D activity recording
• breakpoint generation

Innovation

• Origin-Centric Graph
• Task Breakpoint Generation
• Adaptive Playback
• Virtual Reality
• Spatio-Temporal Scene Graph