Structured and Balanced Multi-component and Multi-layer Neural Networks

2024-06-30

arXiv.org

AI Summary

Efficiently and accurately approximating complex functions—particularly those exhibiting high oscillation and pronounced local features—remains a fundamental challenge in neural approximation. Method: This paper proposes a Structured-Balanced Multi-Component Multi-Layer Neural Network (MMNN). Adopting a “divide-and-conquer” strategy, MMNN adaptively decomposes the target function into multiple subcomponents, each modeled by a dedicated single-layer subnetwork; hierarchical functional decomposition enables synergistic representation across components. Contribution/Results: MMNN introduces a novel balanced architecture that drastically reduces parameter count (by several-fold compared to fully connected networks or standard MLPs) while preserving strong expressive power. It captures local features precisely without requiring explicit regularization. Experiments demonstrate that MMNN significantly outperforms conventional MLPs in training efficiency, approximation accuracy—especially for highly oscillatory functions—and generalization, effectively breaking the traditional accuracy–efficiency trade-off bottleneck.

Technology Category

• Machine Learning: Structured Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

In this work, we propose a balanced multi-component and multi-layer neural network (MMNN) structure to approximate functions with complex features with both accuracy and efficiency in terms of degrees of freedom and computation cost. The main idea is motivated by a multi-component, each of which can be approximated effectively by a single-layer network, and multi-layer decomposition in a"divide-and-conquer"type of strategy to deal with a complex function. While an easy modification to fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs) through the introduction of balanced multi-component structures in the network, MMNNs achieve a significant reduction of training parameters, a much more efficient training process, and a much improved accuracy compared to FCNNs or MLPs. Extensive numerical experiments are presented to illustrate the effectiveness of MMNNs in approximating high oscillatory functions and its automatic adaptivity in capturing localized features.

Problem

• Accurately approximating functions with complex features
• Reducing training parameters and computational cost
• Adapting to localized and oscillatory functions automatically

Innovation

• Balanced multi-component neural network structure
• Multi-layer decomposition for complex features
• Reduced training parameters and improved accuracy