Frequency-Adaptive Sharpness Regularization for Improving 3D Gaussian Splatting Generalization

2025-11-22

AI Summary

3D Gaussian Splatting (3DGS) suffers from overfitting under sparse-view settings, leading to poor generalization in novel-view synthesis and floating artifacts. To address this, we propose Frequency-Adaptive Sharpness Regularization (FASR), which integrates the Sharpness-Aware Minimization (SAM) principle with a local frequency-aware mechanism: it dynamically modulates regularization strength and neighborhood scale according to the spectral characteristics of image regions, effectively suppressing high-frequency noise while preserving salient geometric and textural details. By smoothing the loss landscape at the optimization objective level, FASR enhances model robustness to unseen viewpoints. Extensive experiments across diverse sparse-input configurations and standard benchmarks demonstrate that FASR consistently outperforms state-of-the-art 3DGS baselines, achieving significant improvements in PSNR, SSIM, and LPIPS. Crucially, it balances strong generalization capability with faithful reconstruction of fine-grained geometry and appearance.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite 3D Gaussian Splatting (3DGS) excelling in most configurations, it lacks generalization across novel viewpoints in a few-shot scenario because it overfits to the sparse observations. We revisit 3DGS optimization from a machine learning perspective, framing novel view synthesis as a generalization problem to unseen viewpoints-an underexplored direction. We propose Frequency-Adaptive Sharpness Regularization (FASR), which reformulates the 3DGS training objective, thereby guiding 3DGS to converge toward a better generalization solution. Although Sharpness-Aware Minimization (SAM) similarly reduces the sharpness of the loss landscape to improve generalization of classification models, directly employing it to 3DGS is suboptimal due to the discrepancy between the tasks. Specifically, it hinders reconstructing high-frequency details due to excessive regularization, while reducing its strength leads to under-penalizing sharpness. To address this, we reflect the local frequency of images to set the regularization weight and the neighborhood radius when estimating the local sharpness. It prevents floater artifacts in novel viewpoints and reconstructs fine details that SAM tends to oversmooth. Across datasets with various configurations, our method consistently improves a wide range of baselines. Code will be available at https://bbangsik13.github.io/FASR.

Problem

• Improves 3D Gaussian Splatting generalization across novel viewpoints
• Addresses overfitting to sparse observations in few-shot scenarios
• Prevents floater artifacts while preserving high-frequency image details

Innovation

• Frequency-Adaptive Sharpness Regularization improves 3DGS generalization
• Adapts regularization weight based on local image frequency
• Reduces floater artifacts while preserving high-frequency details