CharGen: Fast and Fluent Portrait Modification

2025-09-29

AI Summary

Addressing the challenge of simultaneously achieving fine-grained control, real-time generation speed, and high visual fidelity in interactive portrait image editing, this paper proposes a lightweight streaming diffusion editing framework. Methodologically: (1) it introduces concept sliders—attribute-decoupled, interpretable controls enabling real-time manipulation of fine-grained semantic attributes (e.g., facial features, expressions, accessories); (2) it integrates StreamDiffusion for low-latency denoising sampling; and (3) it incorporates a lightweight repair step to recover texture details and preserve structural consistency during acceleration. Experiments demonstrate that our method achieves 2–4× faster editing than InstructPix2Pix and Gemini while maintaining identity preservation. It delivers high-quality, responsive real-time portrait editing with enhanced interactivity and precise controllability—significantly improving both user experience and generation fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Interactive editing of character images with diffusion models remains challenging due to the inherent trade-off between fine-grained control, generation speed, and visual fidelity. We introduce CharGen, a character-focused editor that combines attribute-specific Concept Sliders, trained to isolate and manipulate attributes such as facial feature size, expression, and decoration with the StreamDiffusion sampling pipeline for more interactive performance. To counteract the loss of detail that often accompanies accelerated sampling, we propose a lightweight Repair Step that reinstates fine textures without compromising structural consistency. Throughout extensive ablation studies and in comparison to open-source InstructPix2Pix and closed-source Google Gemini, and a comprehensive user study, CharGen achieves two-to-four-fold faster edit turnaround with precise editing control and identity-consistent results. Project page: https://chargen.jdihlmann.com/

Problem

• Addresses slow portrait editing with diffusion models
• Improves attribute control while maintaining visual fidelity
• Accelerates generation speed without losing detail

Innovation

• Concept Sliders isolate and manipulate specific attributes
• StreamDiffusion pipeline enables faster interactive editing
• Lightweight Repair Step reinstates fine textures and details