TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration

2026-08-10

AI Summary

This work addresses the geometric bias and limited reconstruction quality in existing sparse-view 3D Gaussian Splatting (3DGS) methods based on diffusion models, which stem from inconsistencies between training and inference denoising trajectories. To resolve this, the authors propose a strategy trajectory distillation framework that introduces the Learning Using Privileged Information (LUPI) paradigm to this task for the first time. During training, a teacher model leverages privileged geometric information from additional viewpoints to guide a sparse-view student model, aligning the student’s denoising trajectory with the teacher’s output. This alignment harmonizes the denoising direction and cross-view responses between training and inference. Evaluated across multiple datasets and sparse-view settings, the method significantly improves reconstruction fidelity and generalization, outperforming current state-of-the-art approaches.

Abstract

We present TRACE-GS, an on-policy trajectory distillation framework that leverages privileged geometric conditioning at training time, thereby adapting a diffusion prior to sparse-view 3D Gaussian Splatting (3DGS) restoration. Rather than pursuing increasingly sophisticated restoration architectures, we identify a more fundamental limitation shared by existing diffusion-based approaches: supervision at independently noised states does not cover those reached during inference. In sparse-view 3DGS, under-constrained geometry biases denoising from the outset, and the resulting deviations compound along the rollout. TRACE-GS instead performs on-policy trajectory distillation: a teacher conditioned on richer geometry from additional training views supplies targets along the sparse-view student's own rollout, aligning denoising directions and cross-view responses at each visited state. This training-only geometry places TRACE-GS in the learning using privileged information (LUPI) setting. At deployment, only the sparse-view student is retained, and its restored renderings serve as pseudo-observations for 3DGS refinement. To the best of our knowledge, TRACE-GS is the first to derive on-policy supervision from privileged geometry for sparse-view 3DGS restoration, achieving consistent gains and strong generalization across datasets and sparse-view settings.

Problem

• sparse-view 3DGS restoration
• diffusion-based approaches
• on-policy supervision
• geometric conditioning
• trajectory distillation

Innovation

• on-policy distillation
• privileged information
• sparse-view 3D reconstruction
• 3D Gaussian Splatting
• trajectory alignment