Coevolving Representations in Joint Image-Feature Diffusion

2026-04-19

AI Summary

Existing joint image-feature diffusion models rely on fixed semantic representation spaces, which are ill-suited to the dynamic demands of generative tasks and thus limit performance gains. This work proposes CoReDi, a novel framework that enables the first-ever co-evolution of semantic representations and the diffusion process. CoReDi employs a lightweight, learnable linear projection to dynamically map high-dimensional semantic features into the generative space, complemented by stop-gradient operations, feature normalization, and tailored regularization to effectively mitigate feature degradation and enhance the complementarity between semantic features and image latents. Evaluated in both VAE latent-space and pixel-space diffusion settings, CoReDi consistently achieves faster convergence and superior sample quality compared to state-of-the-art methods based on static representation spaces.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Representation Learning
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Joint image-feature generative modeling has recently emerged as an effective strategy for improving diffusion training by coupling low-level VAE latents with high-level semantic features extracted from pre-trained visual encoders. However, existing approaches rely on a fixed representation space, constructed independently of the generative objective and kept unchanged during training. We argue that the representation space guiding diffusion should itself adapt to the generative task. To this end, we propose Coevolving Representation Diffusion (CoReDi), a framework in which the semantic representation space evolves during training by learning a lightweight linear projection jointly with the diffusion model. While naively optimizing this projection leads to degenerate solutions, we show that stable coevolution can be achieved through a combination of stop-gradient targets, normalization, and targeted regularization that prevents feature collapse. This formulation enables the semantic space to progressively specialize to the needs of image synthesis, improving its complementarity with image latents. We apply CoReDi to both VAE latent diffusion and pixel-space diffusion, demonstrating that adaptive semantic representations improve generative modeling across both settings. Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.

Problem

• joint diffusion
• fixed representation space
• semantic features
• generative modeling
• adaptive representations

Innovation

• coevolving representations
• joint diffusion
• adaptive semantic features
• feature collapse prevention
• linear projection learning