RepNet: Tackling spectral bias in deep neural networks via parameter reparameterization

2026-06-15

AI Summary

This work addresses the spectral bias of deep neural networks, which hinders their ability to effectively learn high-frequency or multiscale functions characterized by rapid oscillations. To overcome this limitation, the authors propose a trainable reparameterization mechanism that adaptively adjusts the initial slope scales and breakpoint distributions in the first hidden layer of ReLU and tanh networks by modulating their weights and biases. This enables frequency-adaptive scaling during training and provides quantitative estimates of output amplitude and slope to inform initialization. Integrated within physics-informed neural networks (PINNs) and operator learning frameworks, the method significantly enhances high-frequency prediction accuracy across diverse tasks—including multiscale function approximation, forward and inverse problems for partial differential equations, and operator learning—while incurring only minimal computational overhead.

Abstract

Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNet, a reparameterized DNN model for ReLU and tanh networks designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one- and four-dimensional function approximation, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning, demonstrate that RepNet improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features with slightly additional computational cost. These results indicate that RepNet provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.

Problem

• spectral bias
• high-frequency functions
• multiscale problems
• oscillatory behavior
• deep neural networks

Innovation

• spectral bias
• reparameterization
• multiscale problems
• high-frequency approximation
• physics-informed neural networks