Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers

2026-05-16

AI Summary

Existing representation alignment methods in diffusion Transformers predominantly rely on point-to-point matching, which struggles to capture the intrinsic spatial structural relationships present in vision foundation models, thereby limiting both training efficiency and generation quality. This work proposes sREPA, a structured representation alignment framework that, for the first time, elevates the alignment granularity from individual points to structural levels. By explicitly modeling geometric consistency of relational structures between feature maps, sREPA effectively preserves the spatial topological structure of pretrained features. Integrated within a diffusion Transformer architecture, sREPA significantly accelerates model convergence, enhances training stability, and achieves superior generation quality compared to existing approaches.

Abstract

Recent advances in Diffusion Transformers (DiTs) demonstrate that aligning noisy latent states with well-trained semantic features-as pioneered by Representation Alignment (REPA)-can substantially accelerate training and improve generation fidelity. Subsequent analysis(e.g., iREPA) suggests that these gains arise primarily from transferring spatial structure contained in pre-trained vision representations. However, mostly existing alignment methods employ point-wise matching objectives or rely on implicit architectural tweaks, which fail to explicitly model the spatial relational geometry inherent in vision foundation models. We argue that such element-wise supervision is insufficient to capture the rich spatial topology of visual representations, and that effective alignment for generation should instead be formulated as an explicit structural constraint. To this end, we propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps, rather than merely matching individual feature points. By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence, along with improved sample quality, compared to state-of-the-art alignment strategies. Our code and models will be released.

Problem

• Diffusion Transformers
• Representation Alignment
• Spatial Structure
• Relational Geometry
• Feature Matching

Innovation

• structural alignment
• diffusion transformers
• representation alignment
• spatial relational geometry
• feature map consistency