Lipschitz-aware Linearity Grafting for Certified Robustness

2025-10-28

AI Summary

This work addresses the problem of overly loose local Lipschitz constant estimation in neural network robustness certification. To mitigate approximation error, we propose a Linear Grafting–based method that constructs differentiable, bounded linear surrogates for nonlinear activation functions—without requiring certified training. Our key contribution is the first theoretical characterization, from a Lipschitz perspective, that Linear Grafting fundamentally eliminates the dominant source of approximation error—not merely alleviating unstable neurons. By replacing activations with these tailored linear modules, we obtain tighter local Lipschitz bounds. Experiments demonstrate significant tightening of the ℓ∞-norm local Lipschitz constants and substantial improvements in certified robustness across standard benchmarks. Crucially, our approach achieves these gains without additional training overhead and outperforms existing heuristic methods.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Lipschitz constant is a fundamental property in certified robustness, as smaller values imply robustness to adversarial examples when a model is confident in its prediction. However, identifying the worst-case adversarial examples is known to be an NP-complete problem. Although over-approximation methods have shown success in neural network verification to address this challenge, reducing approximation errors remains a significant obstacle. Furthermore, these approximation errors hinder the ability to obtain tight local Lipschitz constants, which are crucial for certified robustness. Originally, grafting linearity into non-linear activation functions was proposed to reduce the number of unstable neurons, enabling scalable and complete verification. However, no prior theoretical analysis has explained how linearity grafting improves certified robustness. We instead consider linearity grafting primarily as a means of eliminating approximation errors rather than reducing the number of unstable neurons, since linear functions do not require relaxation. In this paper, we provide two theoretical contributions: 1) why linearity grafting improves certified robustness through the lens of the $l_infty$ local Lipschitz constant, and 2) grafting linearity into non-linear activation functions, the dominant source of approximation errors, yields a tighter local Lipschitz constant. Based on these theoretical contributions, we propose a Lipschitz-aware linearity grafting method that removes dominant approximation errors, which are crucial for tightening the local Lipschitz constant, thereby improving certified robustness, even without certified training. Our extensive experiments demonstrate that grafting linearity into these influential activations tightens the $l_infty$ local Lipschitz constant and enhances certified robustness.

Problem

• Reducing approximation errors in neural network verification methods
• Obtaining tighter local Lipschitz constants for certified robustness
• Explaining how linearity grafting improves certified robustness theoretically

Innovation

• Grafting linearity into non-linear activation functions
• Eliminating dominant approximation errors for robustness
• Tightening local Lipschitz constant without certified training