Probing Diffusion Denoising Dynamics for Contrastive Representation Learning

2026-07-09

AI Summary

This work addresses the challenge of efficiently adapting the denoising dynamics of a pre-trained diffusion model for discriminative representation learning while preserving its generative capabilities. The proposed approach treats noisy latent variables at different denoising timesteps as multi-view augmentations of the same image. By freezing the Stable Diffusion backbone and applying lightweight fine-tuning via LoRA, the method introduces, for the first time, a joint optimization framework combining noise-level contrastive learning with reconstruction loss. This enables synergistic learning between generative and discriminative objectives without requiring training from scratch. Experimental results demonstrate strong performance: a linear probe achieves 80.1% top-1 accuracy on ImageNet-1K, and unconditional image generation at 256×256 resolution yields an FID of 5.56.

Abstract

Text-to-image diffusion models exhibit unprecedented generative capability and contain rich intermediate representations that can be useful for discriminative vision tasks. Motivated by this observation, we study a focused question: how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning while preserving its generative behavior under parameter-efficient updates? We present D$^3$CL as an investigation of this question. Our key observation is that noisy latents at different diffusion timesteps can be interpreted as stochastic views of the same underlying image, enabling a contrastive objective to be coupled with the standard denoising reconstruction loss. This formulation provides a simple way to probe the interaction between generative denoising and discriminative representation learning without training from scratch. To keep the adaptation lightweight, we apply LoRA updates to a pretrained Stable Diffusion backbone while freezing the original model parameters. D$^3$CL provides strong empirical evidence that reconstruction and noise-level contrastive objectives can be complementary: on ImageNet-1K, it obtains 80.1% linear-probing accuracy and an FID of 5.56 for $256 \times 256$ unconditional generation. Additional ablations on the design space suggest that the usefulness of diffusion features depends on where and how denoising states are sampled. These results establish D$^3$CL as a parameter-efficient adaptation framework for pretrained diffusion models, showing that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.

Problem

• diffusion models
• contrastive learning
• representation learning
• denoising dynamics
• parameter-efficient adaptation

Innovation

• diffusion models
• contrastive learning
• parameter-efficient adaptation
• denoising dynamics
• LoRA