ColorFlow: Retrieval-Augmented Image Sequence Colorization

2024-12-16

arXiv.org

AI Summary

To address the challenge of preserving character/object identity consistency in automatic colorization of black-and-white image sequences, this paper proposes a retrieval-augmented diffusion coloring framework tailored for industrial animation and manga production. Methodologically, it introduces the first retrieval-augmented paradigm that requires neither ID-specific fine-tuning nor explicit identity embeddings; instead, it employs a dual-branch diffusion architecture leveraging self-attention to achieve context-aware identity matching and cross-frame color transfer. Extensive experiments on our newly constructed benchmark, ColorFlow-Bench, demonstrate that the proposed method significantly outperforms existing approaches, achieving state-of-the-art performance in color accuracy, temporal consistency, and identity fidelity. The source code and pre-trained models are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.To address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: https://zhuang2002.github.io/ColorFlow/.

Problem

• Automate black-and-white image sequence colorization preserving identity
• Improve controllability and identity consistency in colorization
• Develop a robust retrieval-augmented pipeline for industrial applications

Innovation

• Three-stage diffusion-based framework for colorization
• Retrieval Augmented Colorization pipeline with references
• Dual-branch design for identity extraction and colorization