Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing

2025-08-19

AI Summary

This work addresses the challenge of sketch-based structural editing of 3D scenes in video, particularly under large camera motions (e.g., significant rotation or scaling), where maintaining cross-view consistency, preserving unedited regions, and ensuring geometric fidelity from 2D sketches to 3D outputs remain difficult. We propose a 3D-aware video editing framework comprising three key components: (1) dense stereo matching to jointly estimate scene point clouds and camera parameters; (2) a point-cloud-guided geometric editing module coupled with a 3D-aware mask propagation strategy, enabling sparse sketch/mask inputs to be mapped onto depth-consistent 3D components; and (3) integration of first-frame image editing with video diffusion models to synthesize temporally coherent 3D videos. Extensive experiments demonstrate that our method significantly outperforms existing approaches in view consistency, geometric accuracy, and detail realism.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Structured Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Security and Privacy: Large-scale security measurements

Abstract

Recent video editing methods achieve attractive results in style transfer or appearance modification. However, editing the structural content of 3D scenes in videos remains challenging, particularly when dealing with significant viewpoint changes, such as large camera rotations or zooms. Key challenges include generating novel view content that remains consistent with the original video, preserving unedited regions, and translating sparse 2D inputs into realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a sketch-based 3D-aware video editing method to enable detailed local manipulation of videos with significant viewpoint changes. To solve the challenge posed by sparse inputs, we employ image editing methods to generate edited results for the first frame, which are then propagated to the remaining frames of the video. We utilize sketching as an interaction tool for precise geometry control, while other mask-based image editing methods are also supported. To handle viewpoint changes, we perform a detailed analysis and manipulation of the 3D information in the video. Specifically, we utilize a dense stereo method to estimate a point cloud and the camera parameters of the input video. We then propose a point cloud editing approach that uses depth maps to represent the 3D geometry of newly edited components, aligning them effectively with the original 3D scene. To seamlessly merge the newly edited content with the original video while preserving the features of unedited regions, we introduce a 3D-aware mask propagation strategy and employ a video diffusion model to produce realistic edited videos. Extensive experiments demonstrate the superiority of Sketch3DVE in video editing. Homepage and code: http://http://geometrylearning.com/Sketch3DVE/

Problem

• Editing 3D scene structure in videos with viewpoint changes
• Translating sparse 2D sketches into consistent 3D video outputs
• Preserving unedited regions while integrating new 3D components

Innovation

• Sketch-based 3D-aware video editing method
• Dense stereo estimation for point clouds
• 3D-aware mask propagation strategy