Logarithmic Width Suffices for Robust Memorization

2025-02-16

AI Summary

This work investigates the robust memorization capacity of feedforward ReLU neural networks under ℓₚ-norm adversarial perturbations—specifically, whether a network can memorize N input–output pairs with a perturbation radius that does not vanish as N grows. Leveraging constructive network design, geometric covering arguments, information-theoretic lower bounds, and ℓₚ-ball volume estimates, we establish the first tight width bound for robust memorization: Θ(log N) width is both necessary and sufficient. This sharply contrasts with classical (non-robust) memorization, which requires only constant width, revealing a fundamental capacity cost imposed by robustness. Furthermore, we derive matching upper and lower bounds on the achievable robust radius, showing that logarithmic width suffices to attain a robust radius independent of N. Our results provide foundational insights into the interplay between architecture, robustness, and memorization in deep learning theory.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize robustly, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width logarithmic in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).

Problem

• Robust memorization in neural networks
• Width logarithmic in input samples
• Adversarial perturbations with l_p norms

Innovation

• Logarithmic width suffices robust memorization
• Withstands adversarial perturbations effectively
• Bounds established for general l_p norms