SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision

2026-03-25

AI Summary

This work addresses the challenge of optimizing 3D Gaussian splatting under severe camera misalignment, where the local support of Gaussian primitives leads to vanishing gradients and optimization failure. To overcome this, the authors propose shifting the optimization objective to the frequency domain by leveraging globally supported complex sinusoidal features—termed spectral moments—to construct a global attraction basin. A physics-informed frequency annealing strategy is introduced to mitigate gradient vanishing while avoiding high-frequency periodic local minima. By integrating spectral moment supervision with differentiable rendering, the method enables robust tracking of complex deformations. Experiments demonstrate that, even with severely misaligned initializations, the approach consistently outperforms conventional appearance-based tracking methods across diverse deformation parameterizations.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

3D Gaussian Splatting (3DGS) enables real-time, photorealistic novel view synthesis, making it a highly attractive representation for model-based video tracking. However, leveraging the differentiability of the 3DGS renderer "in the wild" remains notoriously fragile. A fundamental bottleneck lies in the compact, local support of the Gaussian primitives. Standard photometric objectives implicitly rely on spatial overlap; if severe camera misalignment places the rendered object outside the target's local footprint, gradients strictly vanish, leaving the optimizer stranded. We introduce SpectralSplats, a robust tracking framework that resolves this "vanishing gradient" problem by shifting the optimization objective from the spatial to the frequency domain. By supervising the rendered image via a set of global complex sinusoidal features (Spectral Moments), we construct a global basin of attraction, ensuring that a valid, directional gradient toward the target exists across the entire image domain, even when pixel overlap is completely nonexistent. To harness this global basin without introducing periodic local minima associated with high frequencies, we derive a principled Frequency Annealing schedule from first principles, gracefully transitioning the optimizer from global convexity to precise spatial alignment. We demonstrate that SpectralSplats acts as a seamless, drop-in replacement for spatial losses across diverse deformation parameterizations (from MLPs to sparse control points), successfully recovering complex deformations even from severely misaligned initializations where standard appearance-based tracking catastrophically fails.

Problem

• vanishing gradient
• 3D Gaussian Splatting
• video tracking
• camera misalignment
• photometric loss

Innovation

• Spectral Moments
• 3D Gaussian Splatting
• Frequency Annealing
• Differentiable Tracking
• Global Optimization