PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion

2026-08-16

AI Summary

This study addresses the insufficient fidelity of controllable text-to-image models in preserving fine-grained structures, particularly at boundaries and within small regions. To overcome this limitation, we propose a pixel-space controllable diffusion framework built upon the PixelDiT backbone. By employing structure-aware injection and modality-specific gated fusion mechanisms, our approach effectively circumvents latent space bottlenecks. Furthermore, a multi-scale pyramid cycle loss is introduced to balance fine structural details with global layout coherence. Experimental results demonstrate that the proposed framework significantly enhances both visual quality and structural fidelity across depth estimation, semantic segmentation, and edge control tasks. Consequently, this method effectively improves the generation of complex details, offering a robust solution for high-precision controllable image synthesis in challenging scenarios requiring strict structural adherence.

Abstract

Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/

Problem

• Text-to-Image Diffusion
• Fine-Grained Condition Fidelity
• Controllable Generation
• Latent Diffusion Limitations
• Structural Fidelity

Innovation

• Pixel-space diffusion
• Structure-Aware Control Injection
• Multi-Scale Pyramid Cycle Loss
• Fine-grained condition fidelity
• Gated fusion