Expressivity of Neural Networks with Random Weights and Learned Biases

2024-07-01

arXiv.org

AI Summary

This work investigates the functional approximation capability of feedforward and recurrent neural networks under the constraint that only biases are optimized while weights remain randomly fixed. Method: Theoretically, we employ covering number analysis to establish that a single-layer feedforward network with learned biases alone can densely approximate any continuous function; similarly, an RNN with bias-only optimization achieves uniform approximation of Lipschitz-continuous dynamical systems. Experimentally, we evaluate this paradigm on multi-task classification (MNIST, CIFAR-10) and chaotic trajectory prediction (Lorenz system). Results: The bias-only training achieves performance comparable to full-parameter optimization. This study provides the first dual theoretical and empirical demonstration that bias plasticity alone—without synaptic weight updates—suffices for universal multi-task approximation, revealing a novel mechanism of neural dynamical plasticity under static synaptic connectivity.

Technology Category

• Machine Learning: Bio-inspired Learning
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Landmark universal function approximation results for neural networks with trained weights and biases provided impetus for the ubiquitous use of neural networks as learning models in Artificial Intelligence (AI) and neuroscience. Recent work has pushed the bounds of universal approximation by showing that arbitrary functions can similarly be learned by tuning smaller subsets of parameters, for example the output weights, within randomly initialized networks. Motivated by the fact that biases can be interpreted as biologically plausible mechanisms for adjusting unit outputs in neural networks, such as tonic inputs or activation thresholds, we investigate the expressivity of neural networks with random weights where only biases are optimized. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can be trained to perform multiple tasks by learning biases only. We further show that an equivalent result holds for recurrent neural networks predicting dynamical system trajectories. Our results are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on multi-task methods such as bias fine-tuning and unit masking.

Problem

• Exploring universal approximation with only learned biases in neural networks
• Investigating fixed random weights' role in function approximation
• Validating bias-based tuning for dynamics in neuroscience and AI

Innovation

• Fixed random weights with learned biases
• Approximates continuous functions dynamically
• Applies to feedforward and recurrent networks