Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

2026-03-24

AI Summary

This work addresses the challenges posed by the non-convex loss landscapes of deep neural networks, which hinder optimization and theoretical interpretability. By adopting a sparse signal processing perspective, the study uncovers an implicit convex structure within the loss landscape of two-layer ReLU neural networks and establishes a rigorous correspondence between these networks and their convex equivalents. Leveraging tools from convex optimization and sparse modeling, the authors propose a convex reformulation-based framework for both analysis and training. This approach not only offers a novel theoretical interpretation of ReLU networks but also enhances training stability, thereby systematically advancing the integration of classical signal processing principles into deep learning.

Technology Category

• Machine Learning: Deep Learning Theory
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Deep neural networks (DNNs), particularly those using Rectified Linear Unit (ReLU) activation functions, have achieved remarkable success across diverse machine learning tasks, including image recognition, audio processing, and language modeling. Despite this success, the non-convex nature of DNN loss functions complicates optimization and limits theoretical understanding. In this paper, we highlight how recently developed convex equivalences of ReLU NNs and their connections to sparse signal processing models can address the challenges of training and understanding NNs. Recent research has uncovered several hidden convexities in the loss landscapes of certain NN architectures, notably two-layer ReLU networks and other deeper or varied architectures. This paper seeks to provide an accessible and educational overview that bridges recent advances in the mathematics of deep learning with traditional signal processing, encouraging broader signal processing applications.

Problem

• non-convexity
• deep neural networks
• ReLU activation
• loss landscape
• optimization

Innovation

• hidden convexity
• ReLU neural networks
• convex equivalence
• sparse signal processing
• loss landscape