Private Learning with Public Feature Conditioning

2026-06-17

AI Summary

This work addresses the challenge of effectively leveraging public non-sensitive features to improve performance in regression tasks under label differential privacy or with partially sensitive features, where existing methods fall short. The authors propose Cond-DP, a novel approach that, for the first time, incorporates the spectral decay properties of public features into differentially private optimization. By constructing a data-driven conditioning matrix that reshapes the optimization landscape, Cond-DP yields a provably faster-converging variant of differentially private stochastic gradient descent (DPSGD). Crucially, this enhancement incurs no additional privacy cost. Empirical evaluations across diverse datasets and model architectures demonstrate that Cond-DP consistently and significantly outperforms current baselines, exhibiting superior practicality and robustness.

Abstract

We study differentially private (DP) regression in settings where each data sample includes public, non-sensitive features -- common in applications such as recommendation and advertising systems. While such label-DP or semi-sensitive-feature settings have been primarily explored in the context of classification, effective approaches for regression remain underexplored. We introduce Cond-DP, a conditioned variant of DPSGD that leverages the structure of public feature matrices to improve optimization under privacy constraints. Motivated by the observation that these public features often exhibit rapidly decaying spectra, Cond-DP incorporates a data-driven conditioning matrix to reshape the optimization landscape and accelerate convergence. We provide convergence guarantees for convex, strongly convex, and non-convex settings, and recover standard DPSGD as a special case when the conditioning matrix is the identity. We show how to construct an effective conditioning matrix for Cond-DP directly from public features, enabling provably faster convergence than DPSGD in private linear regression without incurring additional privacy cost. Empirically, Cond-DP with this conditioning matrix consistently outperforms state-of-the-art baselines across a wide range of datasets and model architectures under label DP, demonstrating strong and robust performance in practice.

Problem

• differentially private regression
• public features
• label DP
• semi-sensitive features
• privacy-preserving machine learning

Innovation

• differentially private regression
• public feature conditioning
• conditioned DPSGD
• spectral decay
• label DP