Differentially Private Iterative Screening Rules for Linear Regression

2025-02-25

AI Summary

This work addresses the challenge of feature screening for $L_1$-regularized linear regression under differential privacy constraints. We propose the first provably $varepsilon$-differentially private iterative screening framework. Unlike existing non-private screening methods, our approach leverages duality gap analysis and a private noise-injection mechanism to design dynamic, adaptive weak screening rules that effectively mitigate “over-screening” induced by injected noise. We provide rigorous theoretical guarantees: the framework satisfies $varepsilon$-differential privacy and preserves screening consistency. Empirical evaluation demonstrates that, under identical privacy budgets, our method achieves significantly better trade-offs between model accuracy and sparsity compared to baseline private approaches. To the best of our knowledge, this is the first solution for high-dimensional feature selection under differential privacy that simultaneously offers strong theoretical privacy and consistency guarantees alongside practical performance.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Linear $L_1$-regularized models have remained one of the simplest and most effective tools in data science. Over the past decade, screening rules have risen in popularity as a way to eliminate features when producing the sparse regression weights of $L_1$ models. However, despite the increasing need of privacy-preserving models for data analysis, to the best of our knowledge, no differentially private screening rule exists. In this paper, we develop the first private screening rule for linear regression. We initially find that this screening rule is too strong: it screens too many coefficients as a result of the private screening step. However, a weakened implementation of private screening reduces overscreening and improves performance.

Problem

• Develops differentially private screening rules
• Addresses overscreening in L1-regularized models
• Enhances privacy in linear regression analysis

Innovation

• Differentially private screening rule
• Weakened implementation reduces overscreening
• Improves performance in linear regression