Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories

2024-12-06

arXiv.org

AI Summary

Existing text-driven 3D editing methods exhibit poor robustness and low consistency when jointly modifying geometry and appearance. This paper proposes a “Perturb–Correct” NeRF editing framework: first, it adaptively perturbs NeRF parameters via local loss surface analysis to obtain a robust initialization; then, it performs joint optimization of geometry, texture, and style by following a diffusion-model-generated, text-guided edit trajectory while enforcing identity-preserving gradient constraints. Our approach introduces the first generative-trajectory-guided editing paradigm, integrating loss-surface-aware parameter perturbation with cross-modal identity preservation. It enables flexible, multi-dimensional editing—including color, material, and shape—while maintaining structural and semantic coherence. Experiments demonstrate significant improvements in geometric deformability and appearance fidelity under full 360° multi-view evaluation, with superior editing quality and inter-consistency compared to state-of-the-art methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Recent advancements in text-based diffusion models have accelerated progress in 3D reconstruction and text-based 3D editing. Although existing 3D editing methods excel at modifying color, texture, and style, they struggle with extensive geometric or appearance changes, thus limiting their applications. To this end, we propose Perturb-and-Revise, which makes possible a variety of NeRF editing. First, we perturb the NeRF parameters with random initializations to create a versatile initialization. The level of perturbation is determined automatically through analysis of the local loss landscape. Then, we revise the edited NeRF via generative trajectories. Combined with the generative process, we impose identity-preserving gradients to refine the edited NeRF. Extensive experiments demonstrate that Perturb-and-Revise facilitates flexible, effective, and consistent editing of color, appearance, and geometry in 3D. For 360{deg} results, please visit our project page: https://susunghong.github.io/Perturb-and-Revise.

Problem

• Enables extensive geometric and appearance changes in 3D editing.
• Proposes Perturb-and-Revise for flexible NeRF editing.
• Uses generative trajectories and identity-preserving gradients for refinement.

Innovation

• Perturb NeRF parameters with random initializations
• Revise NeRF via generative trajectories
• Impose identity-preserving gradients for refinement