How to warm-start your unfolding network

2025-02-03

AI Summary

To address poor generalization and training instability in over-parameterized unrolled networks for compressed sensing reconstruction, this paper proposes the Continuation-based Deep Expansion Continuation (C-DEC) framework. C-DEC introduces continuation—a technique previously unexplored in unrolled networks—into intermediate variable initialization, enabling progressive regularization strength adjustment to jointly optimize network architecture and critical hidden states. It further replaces the conventional ℓ₂ loss with a log-cosh loss to suppress outlier-induced residual interference. Theoretically and empirically, C-DEC significantly smooths the loss landscape, enhancing convergence stability and generalization performance; it achieves superior reconstruction accuracy on real-world image datasets compared to state-of-the-art unrolled methods. Key contributions are: (i) the first application of continuation to warm-start design in unrolled networks; (ii) theoretical and empirical characterization of its optimization-path smoothing and generalization-enhancement mechanisms; and (iii) a novel, end-to-end differentiable, and robust reconstruction paradigm.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We present a new ensemble framework for boosting the performance of overparameterized unfolding networks solving the compressed sensing problem. We combine a state-of-the-art overparameterized unfolding network with a continuation technique, to warm-start a crucial quantity of the said network's architecture; we coin the resulting continued network C-DEC. Moreover, for training and evaluating C-DEC, we incorporate the log-cosh loss function, which enjoys both linear and quadratic behavior. Finally, we numerically assess C-DEC's performance on real-world images. Results showcase that the combination of continuation with the overparameterized unfolded architecture, trained and evaluated with the chosen loss function, yields smoother loss landscapes and improved reconstruction and generalization performance of C-DEC, consistently for all datasets.

Problem

• Enhancing overparameterized unfolding networks
• Improving compressed sensing problem solving
• Optimizing network architecture with continuation technique

Innovation

• Ensemble framework for unfolding networks
• Continuation technique in network architecture
• Log-cosh loss function for training