Shared-Weights Extender and Gradient Voting for Neural Network Expansion

2025-09-23

AI Summary

Neural networks often suffer from neuron deactivation when dynamically expanded, leading to ineffective capacity growth. To address this, we propose the Shared-Weight Expander (SWE) and the Steepest-Gradient Voting Allocator (SVoD). SWE enforces parameter smoothness and inheritance by coupling newly added neurons with existing ones via weight sharing. SVoD dynamically allocates expansion budgets across layers based on gradient magnitude, enabling hierarchical, adaptive expansion in deep networks. Together, SWE and SVoD support end-to-end backpropagation training. Experiments on four benchmark datasets demonstrate that our method significantly mitigates neuron deactivation. Compared to state-of-the-art expansion strategies and baseline models, it consistently improves both accuracy and parameter efficiency. These results validate the architectural scalability and training stability of the proposed approach.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Expanding neural networks during training is a promising way to augment capacity without retraining larger models from scratch. However, newly added neurons often fail to adjust to a trained network and become inactive, providing no contribution to capacity growth. We propose the Shared-Weights Extender (SWE), a novel method explicitly designed to prevent inactivity of new neurons by coupling them with existing ones for smooth integration. In parallel, we introduce the Steepest Voting Distributor (SVoD), a gradient-based method for allocating neurons across layers during deep network expansion. Our extensive benchmarking on four datasets shows that our method can effectively suppress neuron inactivity and achieve better performance compared to other expanding methods and baselines.

Problem

• Preventing newly added neurons from becoming inactive during network expansion
• Allocating neurons effectively across layers in deep network expansion
• Augmenting neural network capacity without retraining from scratch

Innovation

• Shared-Weights Extender prevents new neuron inactivity
• Steepest Voting Distributor allocates neurons using gradients
• Method couples new neurons with existing ones for integration