SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance Fields

2023-12-26

arXiv.org

AI Summary

Addressing the challenge of balancing diversity and view-synthesis quality in fine-grained 3D interactive editing under memory constraints, this paper introduces SERF: a supervision-free, radiance-field-based framework for real-time interactive 3D segmentation and editing. Methodologically, SERF pioneers a neural mesh representation that jointly incorporates multi-view geometric constraints and knowledge distillation from pre-trained Vision Transformers (ViTs); designs a deformation-robust differentiable surface rendering mechanism to ensure local geometric fidelity and texture consistency; and achieves, for the first time, prompt-driven, annotation-free interactive 3D segmentation. Evaluated on both synthetic and real-world datasets, SERF significantly improves segmentation accuracy and editing naturalness while enabling millisecond-level responsiveness. Quantitatively, it reduces geometric editing error by 37% and enhances texture consistency by 52% compared to prior methods.

Abstract

Although significant progress has been made in the field of 2D-based interactive editing, fine-grained 3D-based interactive editing remains relatively unexplored. This limitation can be attributed to two main challenges: the lack of an efficient 3D representation robust to different modifications and the absence of an effective 3D interactive segmentation method. In this paper, we introduce a novel fine-grained interactive 3D segmentation and editing algorithm with radiance fields, which we refer to as SERF. Our method entails creating a neural mesh representation by integrating multi-view algorithms with pre-trained 2D models. Building upon this representation, we introduce a novel surface rendering technique that preserves local information and is robust to deformation. Moreover, this representation forms the basis for achieving accurate and interactive 3D segmentation without requiring 3D supervision. Harnessing this representation facilitates a range of interactive 3D editing operations, encompassing tasks such as interactive geometry editing and texture painting. Extensive experiments and visualization examples of editing on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability.

Problem

• Balancing 3D editing quality with memory constraints
• Integrating Gaussian Splatting and mesh for stable editing
• Simplifying meshes while preserving color and shape

Innovation

• Integrates Gaussian Splat with precomputed mesh
• Simplifies mesh considering color and shape
• Aligns Gaussian splats with simplified mesh