ColorGPT: Leveraging Large Language Models for Multimodal Color Recommendation

2025-08-12

AI Summary

Color recommendation in vector graphic design faces challenges including semantic interpretation difficulty, data scarcity, and multi-constraint collaborative optimization. This paper pioneers the application of pre-trained large language models (LLMs) to color recommendation, proposing an end-to-end palette generation method grounded in multimodal semantic understanding and commonsense reasoning. Our approach introduces a unified color tokenization scheme and a context-aware prompt engineering framework that supports both palette completion and full-palette synthesis. Experimental results demonstrate that our method achieves superior accuracy over state-of-the-art baselines on palette completion tasks. In full-palette generation, it significantly improves color diversity (+23.6%) and semantic similarity (+18.4%). This work establishes a novel paradigm for LLM-driven visual design, bridging linguistic intelligence with perceptual color semantics in a principled, scalable manner.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Colors play a crucial role in the design of vector graphic documents by enhancing visual appeal, facilitating communication, improving usability, and ensuring accessibility. In this context, color recommendation involves suggesting appropriate colors to complete or refine a design when one or more colors are missing or require alteration. Traditional methods often struggled with these challenges due to the complex nature of color design and the limited data availability. In this study, we explored the use of pretrained Large Language Models (LLMs) and their commonsense reasoning capabilities for color recommendation, raising the question: Can pretrained LLMs serve as superior designers for color recommendation tasks? To investigate this, we developed a robust, rigorously validated pipeline, ColorGPT, that was built by systematically testing multiple color representations and applying effective prompt engineering techniques. Our approach primarily targeted color palette completion by recommending colors based on a set of given colors and accompanying context. Moreover, our method can be extended to full palette generation, producing an entire color palette corresponding to a provided textual description. Experimental results demonstrated that our LLM-based pipeline outperformed existing methods in terms of color suggestion accuracy and the distribution of colors in the color palette completion task. For the full palette generation task, our approach also yielded improvements in color diversity and similarity compared to current techniques.

Problem

• Can pretrained LLMs improve color recommendation accuracy?
• How to recommend colors for incomplete palettes effectively?
• Can LLMs generate diverse palettes from text descriptions?

Innovation

• Leveraging pretrained LLMs for color recommendation
• Systematic testing of multiple color representations
• Effective prompt engineering for palette generation