Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting

2026-05-17

AI Summary

This work addresses the non-convex training problem of deep linear neural networks under squared loss by introducing a lifting approach based on the generalized completely positive cone. It establishes, for the first time, an exact convex reformulation whose formulation is independent of both network depth and data size. Through bilinear factorization, rank-constrained semidefinite programming, and complementarity conditions, the original problem is transformed into a low-dimensional convex reconstruction involving only conic constraints, while preserving strict equivalence in optimal values. This result uncovers an intrinsic connection between deep linear network training and completely positive programming, fully encapsulating all sources of non-convexity within the conic constraint. Consequently, it offers a novel theoretical perspective for analyzing the optimization landscape of deep linear models.

Abstract

We show that the training problem of a deep linear neural network under the squared loss admits an exact convex reformulation in a lifted space over a generalized completely positive cone. The reformulation has the same optimal value as the original nonconvex problem and is linear in the lifted variables, with all nonconvexity encoded in the cone constraint. Its ambient lifted dimension depends only on the input and output dimensions, independent of the network depth and the number of data points, and the bottleneck width enters only through scalar constraints. The construction proceeds by reducing the multilayer parameterization to a bilinear factorization, lifting it to a rank-constrained semidefinite program, expressing the rank constraint via a complementarity condition, and applying a completely positive lifting. While the resulting formulation is computationally intractable in general, it gives an exact conic representation of the nonconvexity induced by linear factorization and connects linear neural network training with copositive programming.

Problem

• linear neural networks
• convex reformulation
• completely positive lifting
• nonconvex optimization
• copositive programming

Innovation

• completely positive lifting
• convex reformulation
• linear neural networks
• copositive programming
• rank-constrained semidefinite program