MagicColor: Multi-Instance Sketch Colorization

2025-03-21

AI Summary

Manual per-instance coloring in multi-instance sketch coloring is inefficient and error-prone, while existing generative methods fail due to the absence of multi-instance paired training data. Method: This paper proposes the first end-to-end automatic multi-instance coloring framework built upon diffusion models. It introduces (1) a novel self-play training strategy to mitigate scarcity of multi-instance annotations; (2) an instance-guided mechanism enabling independent, controllable color generation for each object; and (3) an edge-aware fine-grained color-matching loss to enhance chromatic accuracy and structural consistency. Contribution/Results: Evaluated on a newly constructed benchmark, our method significantly outperforms prior approaches, achieving high-quality, style-consistent multi-instance coloring without manual intervention—thereby substantially lowering the barrier to professional-grade sketch coloring.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

We present extit{MagicColor}, a diffusion-based framework for multi-instance sketch colorization. The production of multi-instance 2D line art colorization adheres to an industry-standard workflow, which consists of three crucial stages: the design of line art characters, the coloring of individual objects, and the refinement process. The artists are required to repeat the process of coloring each instance one by one, which is inaccurate and inefficient. Meanwhile, current generative methods fail to solve this task due to the challenge of multi-instance pair data collection. To tackle these challenges, we incorporate three technical designs to ensure precise character detail transcription and achieve multi-instance sketch colorization in a single forward. Specifically, we first propose the self-play training strategy to solve the lack of training data. Then we introduce an instance guider to feed the color of the instance. To achieve accurate color matching, we present fine-grained color matching with edge loss to enhance visual quality. Equipped with the proposed modules, MagicColor enables automatically transforming sketches into vividly-colored images with accurate consistency and multi-instance control. Experiments on our collected datasets show that our model outperforms existing methods regarding chromatic precision. Specifically, our model critically automates the colorization process with zero manual adjustments, so novice users can produce stylistically consistent artwork by providing reference instances and the original line art. Our code and additional details are available at https://yinhan-zhang.github.io/color

Problem

• Automate multi-instance sketch colorization for efficiency
• Overcome lack of multi-instance pair training data
• Ensure precise color matching and detail transcription

Innovation

• Self-play training strategy for data scarcity
• Instance guider for multi-instance color feeding
• Fine-grained color matching with edge loss