Differentially Private Model Merging

2026-04-22

AI Summary

This work addresses the challenge of dynamically satisfying varying differential privacy (DP) requirements during inference without retraining models. To this end, it proposes two training-free post-processing methods—random selection and linear combination—that generate new models meeting arbitrary target DP parameters by fusing pre-trained models representing different privacy-utility trade-offs. As the first systematic study to leverage model fusion for adaptively fulfilling arbitrary DP guarantees, the paper provides rigorous theoretical analysis grounded in Rényi differential privacy and privacy loss distributions. It further proves that linear combination strictly dominates random selection in terms of the privacy-utility trade-off. Empirical evaluations on both synthetic and real-world datasets validate the effectiveness and practicality of the proposed approaches.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In machine learning applications, privacy requirements during inference or deployment time could change constantly due to varying policies, regulations, or user experience. In this work, we aim to generate a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training steps, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post processing techniques, namely random selection and linear combination, to output a final private model for any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions for general problems. In a case study on private mean estimation, we fully characterize the privacy/utility results and theoretically establish the superiority of linear combination over random selection. Empirically, we validate our approach and analyses on several models and both synthetic and real-world datasets.

Problem

• Differential Privacy
• Model Merging
• Privacy Requirements
• Post Processing
• Privacy-Utility Tradeoff

Innovation

• Differentially Private Model Merging
• Post-processing
• Rényi Differential Privacy
• Privacy-Utility Tradeoff
• Linear Combination