🤖 AI Summary
Existing methods struggle to reliably extract strong lottery ticket subnetworks from randomly initialized networks. This work proposes a dual-scoring mechanism that introduces an expanded scoring tensor space to optimize mask selection under fixed sparsity, effectively reformulating sparse structure discovery as an edge-popup optimization problem on a zero-augmented network. The approach preserves the reachability of original masks and substantially reduces sensitivity to sparsity hyperparameters by employing frozen-weight scoring during training alongside fixed-density mask optimization. Experimental results demonstrate that the method significantly outperforms fixed-density Edge-Popup, initialization-based pruning, and retrospective sparse training approaches across multiple benchmarks, while exhibiting strong robustness to variations in sparsity settings.
📝 Abstract
The lottery ticket hypothesis proposes that large random neural networks contain sparse subnetworks that can match the performance of dense models after comparable training. A stronger version asserts that sufficiently overparameterized random networks contain subnetworks that are already accurate before any weight training. Existing theory establishes that such strong lottery tickets exist, but reliable extraction remains difficult. We revisit edge-popup, a frozen-weight score-training method for extracting strong tickets, and identify layerwise sparsity selection as a central bottleneck. We introduce double-scoring, an augmented score-space parameterization that replaces a layerwise sparsity search with optimization over enlarged score tensors. We prove that fixed-density masking in an augmented score space preserves access to all original-coordinate masks, and we show that the resulting method can be interpreted as edge-popup on a zero-augmented network. In controlled experiments, double-scoring substantially improves strong-ticket extraction over fixed-density edge-popup and pruning-at-initialization baselines, improves on the performance of rewound sparse-training topologies, and exhibits markedly lower sensitivity to sparsity hyperparameters. Ablations show that the gain is not merely due to additional trainable score parameters, but is tied to the augmented score-space competition that induces the effective original sparsity.