Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

2026-07-15

AI Summary

This work addresses the high computational cost of video diffusion Transformers, which stems from their long diffusion timesteps and self-attention mechanisms, and observes that existing acceleration methods fail to fully exploit the spatiotemporal correlations inherent in video data. To this end, the study introduces a novel approach that, for the first time, identifies channel-level spatiotemporal redundancy in the latent space and proposes a lightweight channel reuse algorithm to skip redundant computations. A co-designed reconfigurable systolic array architecture is also developed to enable hardware-software co-optimization. Evaluated across three mainstream models, the proposed method achieves up to 5.9× speedup and 16.0× improvement in energy efficiency while preserving generation quality, as evidenced by PSNR values exceeding 17 dB.

Abstract

Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.

Problem

• video diffusion transformers
• self-attention bottleneck
• spatiotemporal correlation
• compute cost
• acceleration

Innovation

• algorithm-hardware co-design
• video diffusion transformers
• latent space correlation
• channel-wise reuse
• systolic array accelerator