DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Merging

2026-08-10

AI Summary

This work addresses the vulnerability of diffusion model merging to latent backdoors introduced via malicious checkpoint injection, a threat difficult to mitigate without prior knowledge of the attack. To counter this, we propose DiffSafeMerge (DSM), a defense framework that leverages only a small amount of unlabeled clean data and attack-agnostic stress probes to score source model modules. DSM selectively attenuates the contribution of suspicious modules by contracting them toward a trusted reference model, all while operating within a prescribed budget for clean denoising loss. Notably, DSM is the first method to effectively suppress backdoor inheritance without requiring details of the attack, while preserving high-quality generation. Extensive experiments across four distinct attacks, two datasets, and 21 target conditions demonstrate that DSM achieves zero worst-case target attack success rate across all random seeds and attains the lowest average FID.

Abstract

Unconditional diffusion checkpoint merging assumes benign sources, yet a compromised public checkpoint can transfer a dormant backdoor while clean generation appears normal. Mitigation is difficult without knowing the compromised source, trigger, or target, and broad sanitization may degrade image quality. We introduce DiffSafeMerge (DSM), which uses a small unlabeled clean set and fixed, attack-agnostic stress probes to score source blocks, shrink suspicious contributions toward a trusted reference, and select attenuation under a clean denoising-loss budget. We evaluate four attacks, two datasets, and 21 target conditions. Intended merging already has zero worst-target ASR in 10 of 14 source cases; DSM preserves these outcomes and records no target match in the remaining four over three seeds, including three with baseline ASR of 48--100\%. Among methods with zero worst-target ASR on both datasets, DSM obtains the lowest case-averaged FID in the matched seed-0 comparison.

Problem

• backdoor inheritance
• diffusion model merging
• model security
• untrusted checkpoints
• clean generation

Innovation

• Diffusion Model Merging
• Backdoor Mitigation
• Stress Probes
• Attack-Agnostic Defense
• Model Sanitization