CoNNect: A Swiss-Army-Knife Regularizer for Pruning of Neural Networks

2025-02-02

AI Summary

To address structural disconnection and layer collapse in neural network pruning, this paper proposes the differentiable CoNNect regularizer, which— for the first time—incorporates graph connectivity constraints into an L₀-norm approximation regularization framework, explicitly preserving input-to-output path connectivity during optimization. Theoretically, CoNNect guarantees that the pruned network retains the largest connected subgraph structure. It supports both structured and unstructured pruning without modifying the base training pipeline. Its graph-based differentiable regularization term is plug-and-play compatible with mainstream pruning frameworks such as DepGraph and LLM-pruner. Experiments across multiple benchmarks demonstrate that CoNNect significantly improves convergence in iterative pruning and boosts the average accuracy of one-shot pruners by 1.2–2.8% at equivalent sparsity levels.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Structured Learning
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an $L_0$-norm constraint. This paper introduces CoNNect, a novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. CoNNect integrates with established pruning strategies and supports both structured and unstructured pruning. We proof that CoNNect approximates $L_0$-regularization, guaranteeing maximally connected network structures while avoiding issues like layer collapse. Numerical experiments demonstrate that CoNNect improves classical pruning strategies and enhances state-of-the-art one-shot pruners, such as DepGraph and LLM-pruner.

Problem

• Neural Network Pruning
• Connectivity Preservation
• Efficiency Enhancement

Innovation

• CoNNect
• Neural Network Pruning
• Efficiency Enhancement