The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

2026-03-30

AI Summary

This work systematically investigates noise-based pretraining as an initialization strategy for implicit neural representations (INRs), revealing that unstructured noise—such as Gaussian or uniform distributions—significantly enhances INR generalization to unseen signals without requiring real data. Moreover, noise endowed with the characteristic 1/|f|^α spectral profile of natural images achieves a superior trade-off between signal fitting and inverse imaging tasks like image denoising. The proposed approach consistently outperforms conventional random initialization across both image and video applications, matching the performance of state-of-the-art data-driven methods while offering a practical prior for low-resource scenarios. These findings provide theoretical and empirical insights into the role of initialization in INRs, demonstrating that carefully designed synthetic noise can serve as an effective inductive bias.

Abstract

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic $1/|f^α|$ spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html

Problem

• Implicit Neural Representations
• Parameter Initialization
• Signal Priors
• Noise Pretraining
• Inverse Imaging

Innovation

• noise pretraining
• implicit neural representations
• signal priors
• spectral structure
• initialization strategy