HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion

2026-05-15

AI Summary

This work addresses the “granularity dilemma” in pixel-space diffusion models, where capturing both global semantics and high-fidelity details remains challenging. To this end, the authors propose HyperDiT, a unified framework that introduces a hyper-connected cross-scale interaction mechanism. This mechanism integrates a novel cross-attention-based multi-scale semantic anchor query, scale-aware rotary positional encoding (SA-RoPE), and Registers modules to effectively align geometric structures and suppress generation artifacts. Leveraging a pretrained vision foundation model, HyperDiT achieves a state-of-the-art FID score of 1.56 on ImageNet at 256×256 resolution, significantly enhancing photorealism and fine-grained detail quality, and establishing a new performance benchmark in pixel-space image synthesis.

Abstract

Pixel-space diffusion models bypass the reconstruction bottleneck of Variational Autoencoders (VAEs) but face a fundamental"granularity dilemma": capturing global semantics favors large patch scales, while generating high-fidelity details demands fine-grained inputs. To address this issue, we propose HyperDiT, a unified framework establishing Hyper-Connected Cross-Scale Interactions to bridge the semantic and pixel manifold. Diverging from injecting semantics by AdaLN, HyperDiT utilizes Cross-Attention mechanisms, enabling fine-grained tokens to query multi-level semantic anchors globally. To resolve the spatial mismatch during multi-scale interactions, we introduce Scale-Aware Rotary Position Embedding (SA-RoPE) to ensure precise geometric alignment among tokens of varying patch sizes. Furthermore, we incorporate Registers to learn the dense semantics from a pretrained Visual Foundation Model (VFM), effectively reducing generation hallucination and artifacts. Extensive experiments demonstrate that HyperDiT achieves state-of-the-art (SoTA) FID of $\mathbf{1.56}$ on ImageNet $256\times256$ directly within the pixel space. By combining the fine-grained stream with semantic guidance, HyperDiT offers a superior paradigm for high-fidelity pixel generation.

Problem

• pixel-space diffusion
• granularity dilemma
• high-fidelity generation
• semantic-pixel alignment
• diffusion models

Innovation

• Hyper-Connected Transformers
• Pixel-Space Diffusion
• Cross-Scale Interaction
• Scale-Aware Rotary Position Embedding
• Registers