Shifting-based Optimizable Linear Relaxations for General Activation Functions

2026-06-18

AI Summary

This work addresses the limited generalizability of existing neural network verification methods, which rely on manually crafted linear relaxations tailored to specific activation functions and thus struggle to accommodate novel ones. To overcome this limitation, the authors propose SLiR, a method that, for the first time, automatically generates tight, optimizable, and formally sound linear upper and lower bounds for arbitrary activation functions using only their Lipschitz constant or a set of critical points. SLiR leverages slope parameterization and an automatic shifting mechanism, eliminating the need for human intervention. Empirical evaluations demonstrate that SLiR substantially improves verification performance across a range of practical activation functions, achieving up to 7.8 times higher success rates in property verification compared to the current state-of-the-art.

Abstract

The use of neural networks (NNs) is rapidly increasing, including in safety- and security-critical domains. To provide formal guarantees about NN behavior, many verification methods rely on optimizable linear relaxations of activation functions. However, existing techniques depend on hand-crafted relaxations for each activation function. Extension to state-of-the-art activation functions therefore requires substantial manual effort. In contrast, our approach SLiR (Shifting-based Linear Relaxations) is broadly applicable, requiring only a Lipschitz constant or a set of critical points. SLiR parameterizes relaxations by their slope and computes the corresponding offset via a shifting procedure that ensures sound upper and lower bounds over the input domain, enabling efficient optimization while maintaining correctness. Our experiments show that SLiR produces tight relaxations across a wide range of practical activation functions and enables verification of up to 7.8x more properties compared to state-of-the-art methods.

Problem

• neural network verification
• activation functions
• linear relaxations
• formal guarantees
• optimizable relaxations

Innovation

• linear relaxation
• activation function
• neural network verification
• shifting-based optimization
• Lipschitz constant