Nearly Optimal Differentially Private ReLU Regression

2025-03-08

AI Summary

This paper studies ReLU regression—a fundamental nonconvex learning problem—under differential privacy (DP). It is the first to achieve private learning without public data, under relaxed $(varepsilon,delta)$-DP, while removing strong boundedness assumptions on feature or label norms and supporting i.i.d. $O(1)$-sub-Gaussian data. We propose DP-MBGLMtron, a one-pass minibatch generalized linear model perceptron algorithm, which integrates tracking attack analysis and sub-Gaussian moment estimation. Theoretically, we establish an excess population risk upper bound of $ ilde{O}(d^2/(N^2varepsilon^2))$, matching our newly derived minimax lower bound $Omega(d^2/(N^2varepsilon^2))$, thereby achieving logarithmic optimality. Empirical results demonstrate superior utility over existing baselines.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we investigate one of the most fundamental nonconvex learning problems, ReLU regression, in the Differential Privacy (DP) model. Previous studies on private ReLU regression heavily rely on stringent assumptions, such as constant bounded norms for feature vectors and labels. We relax these assumptions to a more standard setting, where data can be i.i.d. sampled from $O(1)$-sub-Gaussian distributions. We first show that when $varepsilon = ilde{O}(sqrt{frac{1}{N}})$ and there is some public data, it is possible to achieve an upper bound of $Tilde{O}(frac{d^2}{N^2 varepsilon^2})$ for the excess population risk in $(epsilon, delta)$-DP, where $d$ is the dimension and $N$ is the number of data samples. Moreover, we relax the requirement of $epsilon$ and public data by proposing and analyzing a one-pass mini-batch Generalized Linear Model Perceptron algorithm (DP-MBGLMtron). Additionally, using the tracing attack argument technique, we demonstrate that the minimax rate of the estimation error for $(varepsilon, delta)$-DP algorithms is lower bounded by $Omega(frac{d^2}{N^2 varepsilon^2})$. This shows that DP-MBGLMtron achieves the optimal utility bound up to logarithmic factors. Experiments further support our theoretical results.

Problem

• Relaxing stringent assumptions in private ReLU regression.
• Achieving optimal utility bounds in differential privacy.
• Proposing a one-pass mini-batch algorithm for DP ReLU regression.

Innovation

• Relaxed assumptions for private ReLU regression
• Proposed DP-MBGLMtron algorithm for one-pass learning
• Achieved optimal utility bound with DP-MBGLMtron