CaricHarmony: Contrastive Diffusion Paths for Identity-Preserving Caricature Synthesis

2026-06-11

AI Summary

This work addresses the challenge in sketch-driven caricature generation where identity and shape conditions interfere within diffusion models, often yielding distorted or unexpressive results. The authors formulate this conflict as a conditional signal contamination problem and propose, for the first time, a training-free contrastive diffusion pathway mechanism. This approach processes identity, shape, and fused signals through parallel, uncontaminated paths and employs gradient guidance via a cross-attention-based shape energy function ℰ_shape and an identity-preserving energy function ℰ_id robust to extreme deformations. The method accommodates arbitrary sketch inputs, significantly enhancing both generation quality and efficiency while maintaining strong identity consistency. It achieves a shape CLIP score of 0.8615 (surpassing 0.8450), a user preference rating of 7.81 (outperforming 6.06), and generates each image in under 16 seconds.

Abstract

Sketch-based caricature synthesis suffers from a fundamental failure mode: when identity and shape conditions are combined in diffusion models, they create destructive interference that causes inevitable collapse toward either bland portraits or unrecognizable distortions. We identify the root cause as \emph{condition signal contamination} -- competing probability distributions in the denoising trajectory that make balanced generation impossible. We present CaricHarmony, the first training-free method that explicitly resolves this contamination through parallel uncontaminated diffusion paths. During inference, we maintain three paths: $\mathcal{P}^{\mathrm{i}}$ (pure identity), $\mathcal{P}^{\mathrm{s}}$ (pure shape), and $\mathcal{P}^{\mathrm{i+s}}$ (harmonized output). Novel energy functions operating on cross-attention features provide gradient guidance that steers $\mathcal{P}^{\mathrm{i+s}}$ toward optimal balance: $\mathcal{E}_{\mathrm{shape}}$ ensures sketch fidelity through layout and semantic alignment, while $\mathcal{E}_{\mathrm{id}}$ employs token-level correspondence matching robust to extreme distortions. Unlike DemoCaricature requiring 70 seconds per-identity fine-tuning or CaricatureBooth constrained to Bezier curves, CaricHarmony accepts any sketch format and generates in under 16 seconds. Experiments demonstrate state-of-the-art performance: 0.8615 shape CLIP score (vs. 0.8450) under comparable identity consistency score, with 7.81 overall user preference score (vs. 6.06). Our method fundamentally reconceptualizes the ID-shape conflict as conditioning signal contamination for diffusion models, enabling unprecedented creative control while preserving recognition.

Problem

• caricature synthesis
• identity preservation
• diffusion models
• conditioning conflict
• sketch-based generation

Innovation

• condition signal contamination
• contrastive diffusion paths
• identity-preserving caricature synthesis
• energy-based guidance
• training-free diffusion