Multi-Object Sketch Animation by Scene Decomposition and Motion Planning

2025-03-25

AI Summary

Existing sketch animation methods struggle to model object-aware motion and jointly optimize complex dynamics in multi-object scenes. This paper proposes the first end-to-end, zero-shot framework for generating animated GIFs and short videos from hand-drawn multi-object sketches. Our method first leverages a large language model (LLM) for semantic-driven scene decomposition and coarse motion planning. It then introduces a compositional Score Distillation Sampling (SDS) mechanism, integrating differentiable compositing rendering with a motion-refinement neural network to enable object-level motion modeling and joint optimization. Crucially, the framework requires no training data—animation is synthesized iteratively via SDS, guided solely by text prompts. Experiments demonstrate substantial improvements over prior art in animation quality, motion plausibility, and object independence. Our approach systematically addresses the two core challenges in multi-object sketch animation: object-aware motion modeling and optimization of intricate, interdependent dynamics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Sketch animation, which brings static sketches to life by generating dynamic video sequences, has found widespread applications in GIF design, cartoon production, and daily entertainment. While current sketch animation methods perform well in single-object sketch animation, they struggle in multi-object scenarios. By analyzing their failures, we summarize two challenges of transitioning from single-object to multi-object sketch animation: object-aware motion modeling and complex motion optimization. For multi-object sketch animation, we propose MoSketch based on iterative optimization through Score Distillation Sampling (SDS), without any other data for training. We propose four modules: LLM-based scene decomposition, LLM-based motion planning, motion refinement network and compositional SDS, to tackle the two challenges in a divide-and-conquer strategy. Extensive qualitative and quantitative experiments demonstrate the superiority of our method over existing sketch animation approaches. MoSketch takes a pioneering step towards multi-object sketch animation, opening new avenues for future research and applications. The code will be released.

Problem

• Challenges in multi-object sketch animation motion modeling
• Difficulties in complex motion optimization for sketches
• Lack of training data for multi-object animation

Innovation

• LLM-based scene decomposition and motion planning
• Motion refinement network for complex optimization
• Compositional SDS for iterative optimization