Palette Aligned Image Diffusion

2025-09-02

AI Summary

This work addresses the ambiguity and instability inherent in palette-conditioned text-to-image diffusion models. Methodologically, we propose Palette-Adapter: (1) palettes are encoded as sparse histograms; (2) two tunable scalar parameters—histogram entropy and palette-histogram distance—are introduced to jointly control color fidelity and chromatic diversity; and (3) a negative histogram mechanism is designed to suppress undesired hues without classifier-free guidance, thereby enhancing palette consistency. Our key contributions are the first integration of sparse histogram modeling with dual-scalar control for palette-conditioned generation, and training on a balanced dataset covering both common and rare colors. Experiments demonstrate that our approach significantly outperforms state-of-the-art methods in color fidelity, visual quality, and generation stability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce the Palette-Adapter, a novel method for conditioning text-to-image diffusion models on a user-specified color palette. While palettes are a compact and intuitive tool widely used in creative workflows, they introduce significant ambiguity and instability when used for conditioning image generation. Our approach addresses this challenge by interpreting palettes as sparse histograms and introducing two scalar control parameters: histogram entropy and palette-to-histogram distance, which allow flexible control over the degree of palette adherence and color variation. We further introduce a negative histogram mechanism that allows users to suppress specific undesired hues, improving adherence to the intended palette under the standard classifier-free guidance mechanism. To ensure broad generalization across the color space, we train on a carefully curated dataset with balanced coverage of rare and common colors. Our method enables stable, semantically coherent generation across a wide range of palettes and prompts. We evaluate our method qualitatively, quantitatively, and through a user study, and show that it consistently outperforms existing approaches in achieving both strong palette adherence and high image quality.

Problem

• Addressing ambiguity in palette-conditioned image generation
• Controlling color adherence and variation via histogram parameters
• Enabling suppression of undesired hues for improved palette fidelity

Innovation

• Palette-Adapter conditions diffusion models with color palettes
• Uses histogram entropy and distance for flexible color control
• Negative histogram mechanism suppresses undesired hues