Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork

2026-06-25

AI Summary

This work addresses the significant utility degradation in differentially private neural network training—such as DP-SGD—caused by repeated noise injection in high-dimensional parameter spaces. The authors propose a novel framework that leverages a hypernetwork pretrained on public data to map private data into target model parameters, injecting privacy-preserving noise only once in a low-dimensional data embedding space, thereby avoiding direct perturbation of high-dimensional parameters. By integrating embedding aggregation, a tailored perturbation mechanism, and LoRA fine-tuning, the method substantially improves model performance under a fixed privacy budget. Empirical results demonstrate clear advantages over DP-SGD and other public-data-based baselines, with notably better FID scores on both synthetic tasks and LoRA fine-tuning of diffusion models.

Abstract

Differentially private (DP) training of neural networks is often hindered by the large amount of noise required by gradient-based methods such as DP-SGD, which repeatedly inject high-dimensional noise in parameter space throughout training. In this paper, we propose a new framework for DP learning that avoids iterative optimization in parameter space. Instead of updating the target model using privatized gradients, we employ a hypernetwork trained on public datasets to map a private dataset to the parameters of the target model. Specifically, each example is embedded into a low-dimensional representation, the embeddings are aggregated and perturbed to obtain a DP dataset embedding, and the hypernetwork generates the target model parameters from this noisy embedding. Because privacy noise is injected only once into a low-dimensional dataset representation, our approach can significantly reduce the adverse effect of noise. We theoretically show in a synthetic setting that, under a fixed privacy budget, models produced by our approach achieve higher utility than those trained with DP-SGD. Moreover, we apply our approach to LoRA fine-tuning of diffusion models and show that it achieves lower FID than LoRA models trained with DP-SGD and other public-data-guided methods.

Problem

• differential privacy
• noise reduction
• parameter-space noise
• private learning
• utility degradation

Innovation

• differential privacy
• hypernetwork
• low-dimensional embedding
• non-iterative parameter generation
• DP-SGD alternative