Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

2026-03-02

AI Summary

This work proposes the 3R framework to address the high sensitivity of existing text-to-video (T2V) generation models to input prompts, a limitation that traditional approaches struggle to overcome without costly fine-tuning or complex post-processing, often sacrificing quality, scalability, or usability. The 3R framework uniquely integrates retrieval-augmented generation (RAG), diffusion-based preference optimization, and temporal frame interpolation—without requiring any model training—to significantly enhance the output quality of arbitrary T2V models. By leveraging RAG to extract semantically rich modifiers, refining prompts via a vision-language model, and improving motion coherence through temporal interpolation, the method effectively boosts both static fidelity and temporal consistency in generated videos, thereby underscoring the pivotal role of prompt optimization in T2V synthesis.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts.

Problem

• Text-to-Video Generation
• Prompt Sensitivity
• Video Quality
• Model Fine-tuning
• Post-editing Artifacts

Innovation

• Prompt Optimization
• Retrieval-Augmented Generation (RAG)
• Text-to-Video Generation
• Preference Optimization
• Temporal Consistency