Neural Networks Trained by Weight Permutation are Universal Approximators

2024-07-01

Neural Networks

AI Summary

This paper investigates whether neural networks retain universal approximation capability under a highly constrained training paradigm—where only weight permutations (without numerical updates) are permitted. Method: Focusing on the approximation of one-dimensional continuous functions, the authors develop an analytical framework integrating combinatorial optimization and probabilistic construction, specifically applied to ReLU feedforward networks. Contribution/Results: They provide the first rigorous proof that, for any continuous function on a compact interval and any desired accuracy, there exists a finite sequence of weight permutations enabling the network to approximate the function arbitrarily well. This result fundamentally departs from conventional gradient-based training relying on continuous parameter updates, establishing theoretical feasibility of universal approximation via purely discrete structural operations. It offers novel insights into neural network optimization mechanisms and lays a foundational theoretical basis for hardware-efficient, permutation-only training methods.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Problem

• Proves permutation training enables ReLU networks to approximate functions.
• Validates permutation method's efficiency in regression tasks.
• Suggests permutation training as a tool for analyzing learning behavior.

Innovation

• Permutation-based training without weight modification
• Theoretical guarantee for ReLU network approximation
• Efficient regression with diverse initializations