Approximation theory for 1-Lipschitz ResNets

2025-05-17

AI Summary

Existing 1-Lipschitz residual networks (ResNets) lack rigorous approximation theory, hindering their theoretical foundation in generative modeling, inverse problem solving, and robust classification. Method: We propose a construction based on explicit Euler discretization of negative gradient flows, leveraging a restricted Stone–Weierstrass theorem, norm-constrained linear mappings, and stacked residual blocks. Contributions/Results: We establish the first universal approximation guarantee for 1-Lipschitz ResNets: they are dense in the space of all scalar 1-Lipschitz functions over compact domains, retain approximation capability even at fixed width, and exactly represent piecewise-affine 1-Lipschitz functions. Crucially, our theoretical framework supports end-to-end training using standard optimizers—bridging a critical gap between theory and practical deployment of 1-Lipschitz deep networks.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

1-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone-Weierstrass Theorem, we first show that these 1-Lipschitz ResNets are dense in the set of scalar 1-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1-Lipschitz ResNets, laying a rigorous foundation for their practical use.

Problem

• Studying approximation capabilities of 1-Lipschitz ResNets
• Proving density in scalar 1-Lipschitz functions
• Enabling universal approximation with norm constraints

Innovation

• 1-Lipschitz ResNets use explicit Euler steps
• Leverage Restricted Stone-Weierstrass Theorem for density
• Norm-constrained linear maps enable fixed-width density