DP-SAPF: Saliency-Aware Parameter Fine-tuning of Public Models for Differentially Private Image Synthesis

2026-05-28

AI Summary

This work addresses the high computational cost of full-parameter fine-tuning in differentially private image generation and the limitations of existing low-rank adaptation (LoRA) approaches, which uniformly apply parameter updates across all attention layers and often suffer from noise accumulation, leading to training instability and degraded image quality. To overcome these issues, the paper proposes a saliency-aware parameter-efficient fine-tuning method (DP-SAPF), which introduces gradient-magnitude-driven saliency estimation into differentially private image synthesis for the first time. DP-SAPF selectively applies LoRA only to the most sensitivity-critical parameters, thereby minimizing unnecessary perturbations. Integrated with DP-SGD, this approach significantly enhances both visual quality and statistical fidelity of generated images on four sensitive datasets while reducing computational overhead.

Abstract

Differentially private (DP) image synthesis generates images that preserve the statistical characteristics of a sensitive dataset, enabling sensitive data analysis and usage while providing rigorous guarantees of privacy leakage. Existing methods fine-tune public models using DP Stochastic Gradient Descent (DP-SGD) on sensitive images to generate synthetic images. But full fine-tuning public models on sensitive images is computationally expensive, because current public models typically contain a large number of parameters. Recent work proposes heuristically using Low-Rank Adaptation (LoRA) on all attention-layer parameters of public models to reduce the number of trainable parameters. However, we argue that exhaustive LoRA coverage across all attention-layer parameters is suboptimal in a DP setting, as it leads to noise accumulation and collapse during private training. To address this issue, we propose DP-SAPF, which uses a saliency-aware strategy to identify specific target parameters for LoRA training under DP. DP-SAPF is inspired by the fact that larger gradients signify higher saliency, indicating that these parameters are most critical for the DP learning. Specifically, we feed the sensitive images into public models, compute gradients, and add noise to the gradients to satisfy DP. Then, DP-SAPF identifies the most salient parameters, those exhibiting high gradient magnitudes on sensitive images, for DP fine-tuning. Experiments on four sensitive image datasets show that DP-SAPF improves the utility and fidelity of synthetic images while requiring fewer computational resources than fine-tuning methods without parameter selection.

Problem

• differentially private image synthesis
• parameter fine-tuning
• Low-Rank Adaptation
• noise accumulation
• saliency-aware

Innovation

• differential privacy
• saliency-aware fine-tuning
• LoRA
• image synthesis
• parameter-efficient learning