Uni-Animator: Towards Unified Visual Colorization

2026-02-26

AI Summary

Existing sketch-based colorization methods struggle to handle both image and video tasks within a unified framework, often suffering from inaccurate reference color transfer, loss of high-frequency physical details, and temporal inconsistency in videos. This work proposes the first unified colorization framework based on a Diffusion Transformer (DiT), which enhances reference alignment through instance-aware patch embeddings, improves detail generation via physics-inspired feature guidance, and models spatiotemporal dependencies using a sketch-conditioned dynamic RoPE encoding scheme. The proposed method achieves performance on par with task-specific models in both image and video colorization, significantly advancing detail fidelity and temporal coherence, and represents the first approach to enable high-quality, cross-domain unified colorization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

We propose Uni-Animator, a novel Diffusion Transformer (DiT)-based framework for unified image and video sketch colorization. Existing sketch colorization methods struggle to unify image and video tasks, suffering from imprecise color transfer with single or multiple references, inadequate preservation of high-frequency physical details, and compromised temporal coherence with motion artifacts in large-motion scenes. To tackle imprecise color transfer, we introduce visual reference enhancement via instance patch embedding, enabling precise alignment and fusion of reference color information. To resolve insufficient physical detail preservation, we design physical detail reinforcement using physical features that effectively capture and retain high-frequency textures. To mitigate motion-induced temporal inconsistency, we propose sketch-based dynamic RoPE encoding that adaptively models motion-aware spatial-temporal dependencies. Extensive experimental results demonstrate that Uni-Animator achieves competitive performance on both image and video sketch colorization, matching that of task-specific methods while unlocking unified cross-domain capabilities with high detail fidelity and robust temporal consistency.

Problem

• sketch colorization
• temporal coherence
• color transfer
• physical details
• unified framework

Innovation

• Diffusion Transformer
• sketch colorization
• temporal coherence
• physical detail reinforcement
• dynamic RoPE encoding