SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models

2026-02-21

AI Summary

Diffusion models suffer from slow inference due to their sequential denoising process, and existing caching methods often conflate content with noise while neglecting the spectral evolution dynamics inherent in the generation process. This work proposes a training-free, dynamic cache scheduling framework that, for the first time, incorporates the spectral prior of diffusion—where low-frequency structures emerge early and high-frequency details appear later—into the caching mechanism. We introduce a Spectral-Evolution-Aware (SEA) filter to disentangle content from noise and assess the reusability of intermediate results based on spectral alignment. The resulting content-adaptive acceleration strategy aligns with the model’s intrinsic spectral evolution, achieving state-of-the-art latency-quality trade-offs across diverse visual generation tasks while significantly improving inference efficiency without compromising output quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Diffusion models are a strong backbone for visual generation, but their inherently sequential denoising process leads to slow inference. Previous methods accelerate sampling by caching and reusing intermediate outputs based on feature distances between adjacent timesteps. However, existing caching strategies typically rely on raw feature differences that entangle content and noise. This design overlooks spectral evolution, where low-frequency structure appears early and high-frequency detail is refined later. We introduce Spectral-Evolution-Aware Cache (SeaCache), a training-free cache schedule that bases reuse decisions on a spectrally aligned representation. Through theoretical and empirical analysis, we derive a Spectral-Evolution-Aware (SEA) filter that preserves content-relevant components while suppressing noise. Employing SEA-filtered input features to estimate redundancy leads to dynamic schedules that adapt to content while respecting the spectral priors underlying the diffusion model. Extensive experiments on diverse visual generative models and the baselines show that SeaCache achieves state-of-the-art latency-quality trade-offs.

Problem

• diffusion models
• inference acceleration
• caching
• spectral evolution
• feature reuse

Innovation

• Spectral-Evolution-Aware
• diffusion models
• caching
• inference acceleration
• SEA filter