InstanceAnimator: Multi-Instance Sketch Video Colorization

2026-03-26

AI Summary

This work addresses key limitations in existing sketch-based video colorization methods—namely, restricted user control, inaccurate instance alignment, and poor detail fidelity—particularly in multi-instance scenarios. To overcome these challenges, we propose a diffusion Transformer–based framework for multi-instance sketch video colorization that enables high-fidelity, highly controllable simultaneous coloring through unified canvas guidance, cross-frame instance matching, and adaptive decoupled control. Our core innovations include a canvas-guided mechanism that enhances user freedom, an instance matching strategy that ensures precise alignment across multiple characters, and an adaptive decoupling module that strengthens semantic detail injection. Experimental results demonstrate that our method significantly outperforms state-of-the-art approaches in visual quality, control flexibility, and instance consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Web Mining and Content Analysis: Content-based information diffusion
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose InstanceAnimator, a novel Diffusion Transformer framework for multi-instance sketch video colorization. Existing methods suffer from three core limitations: inflexible user control due to heavy reliance on single reference frames, poor instance controllability leading to misalignment in multi-character scenarios, and degraded detail fidelity in fine-grained regions. To address these challenges, we introduce three corresponding innovations. First, a Canvas Guidance Condition eliminates workflow fragmentation by allowing free placement of reference elements and background, enabling unprecedented user flexibility. Second, an Instance Matching Mechanism resolves misalignment by integrating instance features with the sketches, ensuring precise control over multiple characters. Third, an Adaptive Decoupled Control Module enhances detail fidelity by injecting semantic features from characters, backgrounds, and text conditions into the diffusion process. Extensive experiments demonstrate that InstanceAnimator achieves superior multi-instance colorization with enhanced user control, high visual quality, and strong instance consistency.

Problem

• multi-instance
• sketch video colorization
• user control
• instance controllability
• detail fidelity

Innovation

• Diffusion Transformer
• multi-instance colorization
• instance matching
• canvas guidance
• adaptive decoupled control