Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency

2025-09-25

AI Summary

Existing pruning methods predominantly rely on parameter magnitude or gradients, neglecting neuron activation dynamics, thus struggling to balance computational efficiency with representational diversity. This paper proposes Budgeted Broadcast—a dynamic, activity-aware pruning framework. It defines a local communication budget as the product of long-term activation rate and fan-out count, and introduces a selective-audience balancing mechanism under an encoding entropy constraint, enabling lightweight local controllers to dynamically prune input/output connections. Crucially, this work is the first to jointly integrate information-flow budgeting with entropy regularization for sparse model construction. Empirical evaluation across diverse tasks—including automatic speech recognition (ASR), face recognition, synaptic prediction, and electron microscopy image segmentation—demonstrates that Budgeted Broadcast significantly outperforms state-of-the-art pruning baselines at equivalent sparsity levels; in several cases, it even surpasses dense counterparts. Notably, it achieves new state-of-the-art F1 and PR-AUC scores on electron microscopy segmentation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Most pruning methods remove parameters ranked by impact on loss (e.g., magnitude or gradient). We propose Budgeted Broadcast (BB), which gives each unit a local traffic budget (the product of its long-term on-rate $a_i$ and fan-out $k_i$). A constrained-entropy analysis shows that maximizing coding entropy under a global traffic budget yields a selectivity-audience balance, $logfrac{1-a_i}{a_i}=βk_i$. BB enforces this balance with simple local actuators that prune either fan-in (to lower activity) or fan-out (to reduce broadcast). In practice, BB increases coding entropy and decorrelation and improves accuracy at matched sparsity across Transformers for ASR, ResNets for face identification, and 3D U-Nets for synapse prediction, sometimes exceeding dense baselines. On electron microscopy images, it attains state-of-the-art F1 and PR-AUC under our evaluation protocol. BB is easy to integrate and suggests a path toward learning more diverse and efficient representations.

Problem

• Develops activity-dependent pruning for neural network efficiency
• Balances selectivity-audience tradeoff under traffic budget constraints
• Improves accuracy and representation diversity at matched sparsity

Innovation

• Local traffic budget guides pruning decisions
• Balances selectivity and audience via entropy maximization
• Uses simple local actuators for fan-in or fan-out pruning