SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models

2025-08-09

AI Summary

This work addresses the challenge of animating static sketches—a task traditionally difficult and heavily reliant on professional expertise. We propose the first sketch animation framework integrating LoRA fine-tuning with differentiable Bézier curve optimization. Methodologically, our approach follows a three-stage paradigm: (1) leveraging a pre-trained text-to-video diffusion model, (2) employing Score Distillation Sampling (SDS) for reference-video-driven motion transfer, and (3) parameterizing motion trajectories via differentiable Bézier curves to enable creative motion customization (e.g., “jumping car”). Crucially, we decouple appearance preservation from motion control, ensuring style consistency and editable geometric parameters from a single input sketch. Experiments demonstrate that our method significantly outperforms existing approaches on single-motion customization tasks, faithfully reproducing reference dynamics while strictly preserving the original sketch’s stylistic characteristics.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Motion and Path Planning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Sketching is a uniquely human tool for expressing ideas and creativity. The animation of sketches infuses life into these static drawings, opening a new dimension for designers. Animating sketches is a time-consuming process that demands professional skills and extensive experience, often proving daunting for amateurs. In this paper, we propose a novel sketch animation model SketchAnimator, which enables adding creative motion to a given sketch, like "a jumping car''. Namely, given an input sketch and a reference video, we divide the sketch animation into three stages: Appearance Learning, Motion Learning and Video Prior Distillation. In stages 1 and 2, we utilize LoRA to integrate sketch appearance information and motion dynamics from the reference video into the pre-trained T2V model. In the third stage, we utilize Score Distillation Sampling (SDS) to update the parameters of the Bezier curves in each sketch frame according to the acquired motion information. Consequently, our model produces a sketch video that not only retains the original appearance of the sketch but also mirrors the dynamic movements of the reference video. We compare our method with alternative approaches and demonstrate that it generates the desired sketch video under the challenge of one-shot motion customization.

Problem

• Automating sketch animation for non-professionals
• Customizing motion in sketches using reference videos
• Retaining sketch appearance while applying dynamic movements

Innovation

• Uses LoRA for appearance and motion learning
• Applies Score Distillation Sampling for motion
• Combines Bezier curves with video dynamics