SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching

2025-12-26

AI Summary

To address the challenge non-expert users face in authoring physically plausible content in VR, this paper proposes a sketch-driven real-time 3D content generation framework. The method jointly models structured mid-air hand-drawn sketches—encoding geometric layout—with dynamic gestures—conveying physical parameters such as velocity, force, and direction—to unify shape representation and physical behavior. By tightly integrating VR hand tracking, sketch recognition, physics engine coupling, and a learned gesture-to-physical-parameter mapping, the system enables end-to-end synthesis of rigid-body motion, elastic deformation, and cloth dynamics. Compared to text-based approaches, our method significantly enhances expressive authoring capability and interaction naturalness, enabling novice users to construct interactive, physically consistent dynamic VR scenes within seconds.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Creating physically realistic content in VR often requires complex modeling tools or predefined 3D models, textures, and animations, which present significant barriers for non-expert users. In this paper, we propose SketchPlay, a novel VR interaction framework that transforms humans' air-drawn sketches and gestures into dynamic, physically realistic scenes, making content creation intuitive and playful like drawing. Specifically, sketches capture the structure and spatial arrangement of objects and scenes, while gestures convey physical cues such as velocity, direction, and force that define movement and behavior. By combining these complementary forms of input, SketchPlay captures both the structure and dynamics of user-created content, enabling the generation of a wide range of complex physical phenomena, such as rigid body motion, elastic deformation, and cloth dynamics. Experimental results demonstrate that, compared to traditional text-driven methods, SketchPlay offers significant advantages in expressiveness, and user experience. By providing an intuitive and engaging creation process, SketchPlay lowers the entry barrier for non-expert users and shows strong potential for applications in education, art, and immersive storytelling.

Problem

• Enables intuitive VR content creation through gesture-driven sketching.
• Transforms sketches and gestures into dynamic, physically realistic scenes.
• Lowers entry barriers for non-experts in creating complex physical phenomena.

Innovation

• Uses sketching to capture object structure and spatial arrangement
• Employs gestures to convey physical cues like velocity and force
• Combines sketches and gestures for dynamic, physically realistic scenes