MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers

2026-06-14

AI Summary

This work addresses the redundant computation across timesteps in Diffusion Transformers (DiT) during inference, a challenge exacerbated by the incompatibility of existing token-level caching methods with Mixture-of-Experts (MoE) architectures. To overcome this limitation, the authors propose MoECa, a fine-grained caching framework that refines the caching granularity from the token level to the expert-branch level, enabling cross-timestep feature reuse. MoECa introduces an expert-aware adaptive caching control mechanism and synchronously updates cache states for both the MoE and attention pathways. Experimental results demonstrate that MoECa achieves up to 2.83× inference speedup on various DiT-MoE models while preserving image generation quality with negligible degradation.

Abstract

Diffusion Transformers with Mixture-of-Experts (DiT-MoE) improve model capacity under sparse activation, but diffusion inference is still bottlenecked by redundant computation across timesteps. Existing caching methods mainly operate at the token level, which becomes suboptimal in DiT-MoE because each token update is internally decomposed into multiple routed expert branches. Our analysis shows that cross-timestep redundancy in DiT-MoE is better characterized at the expert-branch level than at the whole-token level. Based on this observation, we propose MoECa, a fine-grained caching framework that performs branch-level feature reuse across timesteps. MoECa further introduces expert-aware adaptive control and synchronized cache updates across MoE and attention paths to maintain stable intermediate states. Experiments on multiple DiT-MoE models show that MoECa consistently achieves a better speed-quality trade-off than prior caching methods, with up to 2.83$\times$ inference speedup and minimal quality degradation.

Problem

• Diffusion Transformers
• Mixture-of-Experts
• redundant computation
• feature reuse
• expert decomposition

Innovation

• Mixture-of-Experts
• Diffusion Transformers
• feature reuse
• branch-level caching
• adaptive control