Residualized Temporal Sparse Autoencoders for Interpreting Diffusion Models

2026-05-26

AI Summary

Existing approaches struggle to model the dynamic activation structures across timesteps in the denoising process of diffusion models. This work proposes a residualized temporal sparse autoencoder that, for the first time, integrates residualized linear dynamics with sparse autoencoding: a linear predictor captures predictable changes between adjacent timesteps, while the residual components are sparsely encoded to learn temporal feature representations beyond linear dynamics. The method enables a structured decomposition of full activation trajectories, allowing each latent variable to be interpreted as a temporally evolving feature. Experiments on Stable Diffusion 1.5 demonstrate that the approach effectively reconstructs activation trajectories and, through ablation studies, spatiotemporal analysis, and intervention experiments, validates its capacity to yield interpretable, temporally structured representations.

Abstract

Text-to-image diffusion models generate images through an iterative denoising process, so internal neural layers produce trajectories of activations rather than single static representations. Sparse autoencoders (SAEs) have recently been used to decompose diffusion activations into interpretable feature directions, but most approaches analyze activations at individual timesteps or condition on time rather than learning directly from full activation trajectories. In this work, we introduce residualized temporal SAEs for diffusion activation trajectories. We collect activations across denoising time, fit linear predictors between neighboring timesteps, and represent each trajectory using an initial activation together with residual components not explained by these linear dynamics. Training an SAE on this residualized representation encourages sparse latents to capture structure beyond what is linearly predictable. The residualized decoder directions can be mapped back into activation space, allowing each latent to be analyzed as a feature trajectory over denoising time. Through reconstruction and ablation studies, spatiotemporal feature analysis, and qualitative steering experiments on Stable Diffusion~1.5, we show that residualized temporal SAEs provide a useful framework for studying temporally structured diffusion activations.

Problem

• diffusion models
• sparse autoencoders
• activation trajectories
• temporal structure
• interpretability

Innovation

• residualized temporal SAEs
• diffusion activation trajectories
• interpretable features
• sparse autoencoders
• denoising dynamics