Towards Precise Scaling Laws for Video Diffusion Transformers

2024-11-25

arXiv.org

AI Summary

Efficient training of video diffusion Transformers under constrained data and compute budgets remains challenging. Method: We conduct systematic empirical analysis, nonlinear regression modeling, and multi-objective optimization under compute constraints to uncover scalable patterns in model behavior. We identify high sensitivity of validation loss to learning rate and batch size, and derive the first unified scaling law jointly predicting model size, hyperparameters, and compute consumption. Contribution/Results: Our scaling formula enables accurate prediction of validation loss across arbitrary scales, facilitating controllable performance–cost trade-offs. Under a 10¹⁰ TFLOPs budget, it reduces inference cost by 40.1% while preserving generation quality. This work establishes a reproducible, generalizable theoretical and practical framework for scaling video generative models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Scalability of ML Systems
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Achieving optimal performance of video diffusion transformers within given data and compute budget is crucial due to their high training costs. This necessitates precisely determining the optimal model size and training hyperparameters before large-scale training. While scaling laws are employed in language models to predict performance, their existence and accurate derivation in visual generation models remain underexplored. In this paper, we systematically analyze scaling laws for video diffusion transformers and confirm their presence. Moreover, we discover that, unlike language models, video diffusion models are more sensitive to learning rate and batch size, two hyperparameters often not precisely modeled. To address this, we propose a new scaling law that predicts optimal hyperparameters for any model size and compute budget. Under these optimal settings, we achieve comparable performance and reduce inference costs by 40.1% compared to conventional scaling methods, within a compute budget of 1e10 TFlops. Furthermore, we establish a more generalized and precise relationship among validation loss, any model size, and compute budget. This enables performance prediction for non-optimal model sizes, which may also be appealed under practical inference cost constraints, achieving a better trade-off.

Problem

• Video Diffusion Transformers
• Optimization
• Visual Generation Models

Innovation

• Video Diffusion Transformer
• Optimal Setting Prediction
• Cost-Performance Balance