Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration

2025-11-05

AI Summary

This work addresses the challenge of efficiently controlling diffusion model generation quality and semantic alignment without fine-tuning. We propose a black-box optimization method for prompt embedding spaces based on the separable covariance matrix adaptation evolution strategy (sep-CMA-ES), eliminating gradient dependence. Our approach jointly leverages the LAION aesthetic predictor and CLIPScore to construct a weighted fitness function, enabling end-to-end prompt optimization on Stable Diffusion XL Turbo. Experiments on a subset of Parti Prompts demonstrate statistically significant improvements over Adam-based optimization in both aesthetic score and CLIPScore, validating sep-CMA-ES’s superior global search capability and controllability in high-dimensional, discrete prompt spaces. The core contribution is the first systematic application of sep-CMA-ES to diffusion model prompt engineering, establishing a novel, training-free, and highly robust paradigm for generative control.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Deep generative models, especially diffusion architectures, have transformed image generation; however, they are challenging to control and optimize for specific goals without expensive retraining. Embedding Space Exploration, especially with Evolutionary Algorithms (EAs), has been shown to be a promising method for optimizing image generation, particularly within Diffusion Models. Therefore, in this work, we study the performance of an evolutionary optimization method, namely Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES), against the widely adopted Adaptive Moment Estimation (Adam), applied to Stable Diffusion XL Turbo's prompt embedding vector. The evaluation of images combines the LAION Aesthetic Predictor V2 with CLIPScore into a weighted fitness function, allowing flexible trade-offs between visual appeal and adherence to prompts. Experiments on a subset of the Parti Prompts (P2) dataset showcase that sep-CMA-ES consistently yields superior improvements in aesthetic and alignment metrics in comparison to Adam. Results indicate that the evolutionary method provides efficient, gradient-free optimization for diffusion models, enhancing controllability without the need for fine-tuning. This study emphasizes the potential of evolutionary methods for embedding space exploration of deep generative models and outlines future research directions.

Problem

• Optimizing image generation in diffusion models without expensive retraining
• Comparing evolutionary algorithms against Adam for embedding space exploration
• Enhancing controllability through gradient-free optimization of prompt embeddings

Innovation

• Evolutionary optimization outperforms Adam in embedding exploration
• Uses sep-CMA-ES algorithm on Stable Diffusion XL Turbo
• Combines aesthetic and alignment metrics in fitness function