Point'n Move: Interactive Scene Object Manipulation on Gaussian Splatting Radiance Fields

2023-11-28

arXiv.org

AI Summary

This work addresses the challenge of interactive 3D scene editing in Gaussian Splatting Radiance Fields without retraining. Methodologically, it introduces a novel two-stage self-prompted segmentation algorithm that maps 2D user clicks to 3D object masks, followed by mask refinement and Gaussian parameter fusion to enable minimal-disturbance initialization. It further proposes an explicit Gaussian editing mechanism coupled with real-time neural rendering-based hole-filling, jointly handling exposed regions under both forward and full 360° viewpoints. Experiments demonstrate millisecond-level, intuitive point-and-edit interaction; the method achieves superior reconstruction fidelity compared to state-of-the-art object removal approaches, while preserving high-quality radiance field rendering, strong edit flexibility, and computational efficiency.

Abstract

We propose Point'n Move, a method that achieves interactive scene object manipulation with exposed region inpainting. Interactivity here further comes from intuitive object selection and real-time editing. To achieve this, we adopt Gaussian Splatting Radiance Field as the scene representation and fully leverage its explicit nature and speed advantage. Its explicit representation formulation allows us to devise a 2D prompt points to 3D mask dual-stage self-prompting segmentation algorithm, perform mask refinement and merging, minimize change as well as provide good initialization for scene inpainting and perform editing in real-time without per-editing training, all leads to superior quality and performance. We test our method by performing editing on both forward-facing and 360 scenes. We also compare our method against existing scene object removal methods, showing superior quality despite being more capable and having a speed advantage.

Problem

• Interactive scene object manipulation with inpainting
• Real-time editing without per-editing training
• Superior quality and speed in object removal

Innovation

• Gaussian Splatting Radiance Field scene representation
• 2D-to-3D self-prompting segmentation algorithm
• Real-time editing without per-editing training