IReNe: Instant Recoloring of Neural Radiance Fields

2024-05-30

Computer Vision and Pattern Recognition

AI Summary

To address high interaction latency, ambiguous object boundaries, and multi-view inconsistency in NeRF-based color editing, this paper proposes the first single-image-guided recoloring method enabling sub-5-second interactive editing. Our approach introduces three key innovations: (1) an automatic neuron classification mechanism that distinguishes diffusion- from view-dependent neurons, enabling selective fine-tuning of only the final-layer diffusion neurons; (2) a trainable semantic segmentation module embedded into the NeRF pipeline to precisely constrain object boundaries; and (3) a lightweight hierarchical fine-tuning framework built upon pre-trained NeRFs, incorporating parameter freezing and efficient backpropagation. Evaluated on a newly constructed editing benchmark, our method achieves 5–500× speedup over state-of-the-art approaches while significantly improving PSNR, SSIM, and LPIPS. The generated results exhibit high photorealism, accurate boundary fidelity, and strong cross-view consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Advances in NERFs have allowed for 3D scene reconstructions and novel view synthesis. Yet, efficiently editing these representations while retaining photo realism is an emerging challenge. Recent methods face three primary limitations: they're slow for interactive use, lack precision at object boundaries, and struggle to ensure multi-view consistency. We introduce IReNe to address these limitations, enabling swift, near real-time color editing in NeRF. Leveraging a pre-trained NeRF model and a single training image with user-applied color edits, IReNe swiftly adjusts network parameters in seconds. This adjustment allows the model to generate new scene views, accurately representing the color changes from the training image while also controlling object boundaries and view-specific effects. Object boundary control is achieved by integrating a trainable segmentation module into the model. The process gains efficiency by retraining only the weights of the last network layer. We observed that neurons in this layer can be classified into those responsible for view-dependent appearance and those contributing to diffuse appearance. We introduce an automated classification approach to identify these neuron types and exclusively fine-tune the weights of the diffuse neurons. This further accelerates training and ensures consistent color edits across different views. A thorough validation on a new dataset, with edited object colors, shows significant quantitative and qualitative advancements over competitors, accelerating speeds by 5x to 500x.

Problem

• Efficient real-time color editing in NeRF
• Precise control at object boundaries
• Ensuring multi-view consistency in edits

Innovation

• Real-time color editing in NeRF
• Trainable segmentation for boundary control
• Selective fine-tuning of diffuse neurons