SGSST: Scaling Gaussian Splatting StyleTransfer

2024-12-04

arXiv.org

AI Summary

This work addresses the challenging problem of end-to-end style transfer for 3D Gaussian Splatting (3DGS) scenes. To overcome geometric distortion and stylistic inconsistency, we propose the first optimization-driven method tailored for ultra-high-resolution 3D scenes—introducing a Multi-Scale Synchronized Optimization Loss (SOS) grounded in global neural statistics. SOS jointly models geometric fidelity and artistic style without fine-tuning the 3DGS model; instead, it directly optimizes Gaussian parameters, preserving structural integrity and stylistic expressiveness even in multi-megapixel 3D environments. Extensive qualitative, quantitative, and perceptual evaluations demonstrate significant superiority over existing approaches. Our method achieves, for the first time, full-scene, high-resolution 3D style transfer—enabling precise, controllable content generation in neural rendering. This establishes a novel paradigm for geometry-aware, style-consistent 3D scene editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Applying style transfer to a full 3D environment is a challenging task that has seen many developments since the advent of neural rendering. 3D Gaussian splatting (3DGS) has recently pushed further many limits of neural rendering in terms of training speed and reconstruction quality. This work introduces SGSST: Scaling Gaussian Splatting Style Transfer, an optimization-based method to apply style transfer to pretrained 3DGS scenes. We demonstrate that a new multiscale loss based on global neural statistics, that we name SOS for Simultaneously Optimized Scales, enables style transfer to ultra-high resolution 3D scenes. Not only SGSST pioneers 3D scene style transfer at such high image resolutions, it also produces superior visual quality as assessed by thorough qualitative, quantitative and perceptual comparisons.

Problem

• Apply style transfer to pretrained 3D Gaussian splatting scenes
• Enable ultra-high resolution 3D scene style transfer
• Improve visual quality via multiscale loss optimization

Innovation

• Uses 3D Gaussian splatting for style transfer
• Introduces multiscale loss with SOS optimization
• Enables ultra-high resolution 3D scene styling