Target noise: A pre-training based neural network initialization for efficient high resolution learning

2026-02-06

AI Summary

This work proposes a self-supervised pretraining-based initialization strategy to address the limitations of conventional neural network initialization methods such as Xavier and Kaiming, which rely on random sampling and neglect structural information relevant to the optimization process. These traditional approaches often lead to slow convergence and pronounced low-frequency bias in high-resolution tasks. The proposed method uniquely leverages white noise as a self-supervised signal to generate structured initial weights by fitting random noise, requiring neither additional data nor architectural modifications. This approach effectively mitigates the low-frequency bias inherent in Implicit Neural Representations (INRs) and Deep Image Priors (DIP), significantly accelerating convergence and enhancing stability in high-resolution settings while enabling earlier capture of high-frequency components for more efficient optimization.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Weight initialization plays a crucial role in the optimization behavior and convergence efficiency of neural networks. Most existing initialization methods, such as Xavier and Kaiming initializations, rely on random sampling and do not exploit information from the optimization process itself. We propose a simple, yet effective, initialization strategy based on self-supervised pre-training using random noise as the target. Instead of directly training the network from random weights, we first pre-train it to fit random noise, which leads to a structured and non-random parameter configuration. We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture. The proposed method is particularly effective for implicit neural representations (INRs) and Deep Image Prior (DIP)-style networks, which are known to exhibit a strong low-frequency bias during optimization. After noise-based pre-training, the network is able to capture high-frequency components much earlier in training, leading to faster and more stable convergence. Although random noise contains no semantic information, it serves as an effective self-supervised signal (considering its white spectrum nature) for shaping the initialization of neural networks. Overall, this work demonstrates that noise-based pre-training offers a lightweight and general alternative to traditional random initialization, enabling more efficient optimization of deep neural networks.

Problem

• weight initialization
• high-frequency learning
• implicit neural representations
• convergence efficiency
• low-frequency bias

Innovation

• noise-based pre-training
• neural network initialization
• self-supervised learning
• implicit neural representations
• high-frequency learning