🤖 AI Summary
To address the challenges of jointly identifying efficient subnetworks and “lottery ticket” subnetworks in supernets—namely, low efficiency, cumbersome multi-stage pipelines, and poor cross-task generalization—this paper proposes SuperTickets, the first framework enabling end-to-end differentiable co-optimization of neural architecture search (NAS) and parameter pruning. Its core innovations include dynamic connectivity modeling and progressive sparsity optimization, which jointly learn subnetwork topology and pruning masks in a unified differentiable training process—eliminating the conventional search-train-prune-retrain cycle. Extensive experiments across three vision tasks and four benchmark datasets demonstrate that SuperTickets consistently outperforms state-of-the-art NAS and pruning methods, achieving higher accuracy with reduced computational cost. Moreover, the discovered lottery-ticket subnetworks exhibit strong cross-task transferability: they maintain superior performance without task-specific retraining.
📝 Abstract
Neural architecture search (NAS) has demonstrated amazing success in searching for efficient deep neural networks (DNNs) from a given supernet. In parallel, the lottery ticket hypothesis has shown that DNNs contain small subnetworks that can be trained from scratch to achieve a comparable or higher accuracy than original DNNs. As such, it is currently a common practice to develop efficient DNNs via a pipeline of first search and then prune. Nevertheless, doing so often requires a search-train-prune-retrain process and thus prohibitive computational cost. In this paper, we discover for the first time that both efficient DNNs and their lottery subnetworks (i.e., lottery tickets) can be directly identified from a supernet, which we term as SuperTickets, via a two-in-one training scheme with jointly architecture searching and parameter pruning. Moreover, we develop a progressive and unified SuperTickets identification strategy that allows the connectivity of subnetworks to change during supernet training, achieving better accuracy and efficiency trade-offs than conventional sparse training. Finally, we evaluate whether such identified SuperTickets drawn from one task can transfer well to other tasks, validating their potential of handling multiple tasks simultaneously. Extensive experiments and ablation studies on three tasks and four benchmark datasets validate that our proposed SuperTickets achieve boosted accuracy and efficiency trade-offs than both typical NAS and pruning pipelines, regardless of having retraining or not. Codes and pretrained models are available at https://github.com/RICE-EIC/SuperTickets.
