On the Lipschitz Continuity of Set Aggregation Functions and Neural Networks for Sets

2025-05-30

AI Summary

This work systematically investigates the Lipschitz continuity of permutation-invariant aggregation functions—such as sum, mean, and max—in set neural networks. Addressing three canonical unordered multiset distances (Hausdorff, optimal transport, and symmetric difference), we establish the first rigorous characterization of Lipschitz conditions for each aggregation function, revealing that each satisfies Lipschitz continuity with respect to exactly one of these distances and deriving its precise Lipschitz constant. Building upon this, we propose the first theoretical framework for bounding the global Lipschitz constant of set neural networks, yielding an explicit, compositional upper bound. Our analysis uncovers a fundamental alignment between aggregation mechanisms and distance metrics, demonstrating that robustness to input perturbations and generalization under distribution shift are inherently governed by this matching. Extensive experiments validate that the derived bound effectively quantifies model robustness and generalization performance.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Agreement, Argumentation & Negotiation
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The Lipschitz constant of a neural network is connected to several important properties of the network such as its robustness and generalization. It is thus useful in many settings to estimate the Lipschitz constant of a model. Prior work has focused mainly on estimating the Lipschitz constant of multi-layer perceptrons and convolutional neural networks. Here we focus on data modeled as sets or multisets of vectors and on neural networks that can handle such data. These models typically apply some permutation invariant aggregation function, such as the sum, mean or max operator, to the input multisets to produce a single vector for each input sample. In this paper, we investigate whether these aggregation functions are Lipschitz continuous with respect to three distance functions for unordered multisets, and we compute their Lipschitz constants. In the general case, we find that each aggregation function is Lipschitz continuous with respect to only one of the three distance functions. Then, we build on these results to derive upper bounds on the Lipschitz constant of neural networks that can process multisets of vectors, while we also study their stability to perturbations and generalization under distribution shifts. To empirically verify our theoretical analysis, we conduct a series of experiments on datasets from different domains.

Problem

• Study Lipschitz continuity of set aggregation functions
• Compute Lipschitz constants for multiset distance functions
• Analyze stability and generalization of set-based neural networks

Innovation

• Analyzes Lipschitz continuity of set aggregation functions
• Derives bounds for neural networks processing multisets
• Tests stability and generalization under distribution shifts