Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

2025-02-26

AI Summary

This paper addresses the challenge of effectively modeling high-frequency signals with neural networks. We propose the Fourier Multi-Component Multi-Layer Neural Network (FMMNN), which jointly designs network architecture and Fourier-basis activation functions to enable frequency-domain multi-component decomposition and multi-layer nonlinear superposition. Theoretically, we establish, for the first time, that FMMNN achieves exponential approximation rates and demonstrate its optimization landscape is provably superior to that of standard fully connected networks. Empirically, FMMNN consistently outperforms existing methods in high-frequency-dominant tasks—including image reconstruction and partial differential equation solving—delivering both higher accuracy and computational efficiency. The core innovation lies in deeply embedding frequency-domain priors into both the network architecture and activation mechanism, yielding a novel paradigm for high-frequency signal modeling that bridges rigorous theoretical guarantees with practical performance.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Computational social science
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The two most critical ingredients of a neural network are its structure and the activation function employed, and more importantly, the proper alignment of these two that is conducive to the effective representation and learning in practice. In this work, we introduce a surprisingly effective synergy, termed the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), and demonstrate its surprising adaptability and efficiency in capturing high-frequency components. First, we theoretically establish that FMMNNs have exponential expressive power in terms of approximation capacity. Next, we analyze the optimization landscape of FMMNNs and show that it is significantly more favorable compared to fully connected neural networks. Finally, systematic and extensive numerical experiments validate our findings, demonstrating that FMMNNs consistently achieve superior accuracy and efficiency across various tasks, particularly impressive when high-frequency components are present.

Problem

• Enhance neural network efficiency and accuracy.
• Capture high-frequency components effectively.
• Optimize neural network structure and activation.

Innovation

• Fourier Multi-Component Neural Networks
• Exponential expressive power
• Superior accuracy efficiency