StylizedGS: Controllable Stylization for 3D Gaussian Splatting

2024-04-08

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Existing NeRF-based 3D stylization methods suffer from inefficient inference, inaccurate geometric style transfer, and limited artistic controllability. This paper proposes the first 3D Gaussian Splatting–based, single-reference-image-driven neural style transfer framework for efficient, geometry-consistent, and perceptually adjustable 3D stylization. We introduce three key innovations: (i) filter-based floating-point elimination to enhance numerical stability; (ii) a nearest-neighbor style loss for faithful texture reproduction; and (iii) depth-preserving regularization for geometric fidelity. These enable fine-grained, color-, scale-, and region-level controllable editing. By integrating multi-objective differentiable optimization with post-filtering refinement, our method significantly improves stroke reconstruction quality and geometric accuracy across diverse scenes and styles. Inference is accelerated by 1–2 orders of magnitude over NeRF, enabling high-fidelity rendering and real-time interactive potential.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As XR technology continues to advance rapidly, 3D generation and editing are increasingly crucial. Among these, stylization plays a key role in enhancing the appearance of 3D models. By utilizing stylization, users can achieve consistent artistic effects in 3D editing using a single reference style image, making it a user-friendly editing method. However, recent NeRF-based 3D stylization methods encounter efficiency issues that impact the user experience, and their implicit nature limits their ability to accurately transfer geometric pattern styles. Additionally, the ability for artists to apply flexible control over stylized scenes is considered highly desirable to foster an environment conducive to creative exploration. To address the above issues, we introduce StylizedGS, an efficient 3D neural style transfer framework with adaptable control over perceptual factors based on 3D Gaussian Splatting (3DGS) representation. We propose a filter-based refinement to eliminate floaters that affect the stylization effects in the scene reconstruction process. The nearest neighbor-based style loss is introduced to achieve stylization by fine-tuning the geometry and color parameters of 3DGS, while a depth preservation loss with other regularizations is proposed to prevent the tampering of geometry content. Moreover, facilitated by specially designed losses, StylizedGS enables users to control color, stylized scale, and regions during the stylization to possess customization capabilities. Our method achieves high-quality stylization results characterized by faithful brushstrokes and geometric consistency with flexible controls. Extensive experiments across various scenes and styles demonstrate the effectiveness and efficiency of our method concerning both stylization quality and inference speed.

Problem

• Efficient 3D neural style transfer with controllable parameters
• Eliminating floaters that degrade stylization effects in reconstruction
• Enabling flexible user control over color, scale and regions

Innovation

• 3D Gaussian Splatting representation for style transfer
• Filter-based refinement to eliminate floaters
• Nearest neighbor style loss with depth preservation