MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation

2025-09-30

AI Summary

Insufficient motion realism in image-to-video generation stems primarily from the difficulty of universally modeling physical constraints, object interactions, and domain-specific dynamics. To address this, we propose a retrieval-augmented, context-aware motion transfer framework. Our method employs a video encoder and a dedicated resampler to extract high-order motion priors, leverages a causal Transformer for motion-context learning, and introduces an attention-injection-based motion adapter that seamlessly integrates retrieved reference motion into a pre-trained diffusion model. The framework enables zero-shot cross-domain generalization; its modular motion database is plug-and-play and requires no fine-tuning. Extensive experiments demonstrate significant improvements in motion fidelity across multiple foundation models and diverse scenarios, with negligible increase in inference overhead. The approach thus achieves both high efficiency and strong generalization capability.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Motion and Path Planning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Image-to-video generation has made remarkable progress with the advancements in diffusion models, yet generating videos with realistic motion remains highly challenging. This difficulty arises from the complexity of accurately modeling motion, which involves capturing physical constraints, object interactions, and domain-specific dynamics that are not easily generalized across diverse scenarios. To address this, we propose MotionRAG, a retrieval-augmented framework that enhances motion realism by adapting motion priors from relevant reference videos through Context-Aware Motion Adaptation (CAMA). The key technical innovations include: (i) a retrieval-based pipeline extracting high-level motion features using video encoder and specialized resamplers to distill semantic motion representations; (ii) an in-context learning approach for motion adaptation implemented through a causal transformer architecture; (iii) an attention-based motion injection adapter that seamlessly integrates transferred motion features into pretrained video diffusion models. Extensive experiments demonstrate that our method achieves significant improvements across multiple domains and various base models, all with negligible computational overhead during inference. Furthermore, our modular design enables zero-shot generalization to new domains by simply updating the retrieval database without retraining any components. This research enhances the core capability of video generation systems by enabling the effective retrieval and transfer of motion priors, facilitating the synthesis of realistic motion dynamics.

Problem

• Generating videos with realistic motion from images
• Modeling complex physical constraints and object interactions
• Generalizing motion dynamics across diverse scenarios

Innovation

• Retrieval-based pipeline extracts semantic motion features
• Causal transformer enables in-context motion adaptation learning
• Attention-based adapter injects motion into diffusion models