PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion

2026-06-26

AI Summary

Existing end-to-end pixel-level diffusion models struggle to jointly model low-frequency semantics and high-frequency details in high-dimensional spaces. This work proposes PixelU, an extremely simple single-stage U-shaped diffusion Transformer that efficiently propagates high-frequency information through zero-overhead skip connections and constructs a low-frequency semantic manifold via constant-channel spatial downsampling. The approach reveals the redundancy of conventional pixel decoders under the x-prediction paradigm and introduces a novel frequency-decoupled framework. PixelU achieves state-of-the-art performance with FID scores of 1.63 and 1.92 on ImageNet at resolutions of 256×256 and 512×512, respectively, surpassing the strong JiT-G baseline at approximately one-third of its computational cost.

Abstract

End-to-end pixel-space diffusion models bypass the lossy compression of Latent Diffusion Models (LDMs) but struggle to jointly model low-frequency semantics and high-frequency signals in high-dimensional space. Existing works heavily rely on complex pixel decoders to alleviate this issue. In this paper, we challenge this trend by revealing that these decoders primarily compensate for the optimization difficulties inherent to velocity prediction ($v$-prediction). Under the clean data paradigm ($x$-prediction), they are redundant. Motivated by this insight, we advocate for simplicity over complexity and introduce PixelU, a minimalist, single-stage U-shaped Diffusion Transformer tailored for pixel space. PixelU abandons auxiliary decoders in favor of zero-cost skip connections, which provide an "information highway" that directly routes uncorrupted high-frequency spatial details from shallow to deep layers. To further enable the backbone to focus exclusively on modeling low-frequency semantics, we introduce a constant-channel spatial down-sampling mechanism as a natural low-pass filter, which compresses deep features into a compact, low-frequency semantic manifold. Extensive experiments demonstrate that this decoupling of frequencies could outperform the strong baseline (JiT-G) with only about 1/3 of its computation cost. On ImageNet 256$\times$256 and 512$\times$512, PixelU achieves FID of 1.63 and 1.92 respectively, surpassing recent pixel-space methods and establishing a simple yet powerful new paradigm for end-to-end diffusion models.

Problem

• pixel-space diffusion
• frequency modeling
• low-frequency semantics
• high-frequency signals
• end-to-end diffusion

Innovation

• Pixel Diffusion
• U-Shaped Transformer
• Frequency Decoupling
• Skip Connections
• Spatial Downsampling