Tracking High-order Evolutions via Cascading Low-rank Fitting

2026-04-13

AI Summary

This work addresses the inefficiency of conventional high-order diffusion models, which independently model each derivative order, leading to linear growth in parameters and computational cost with respect to the order. To overcome this limitation, the authors propose a cascaded low-rank approximation mechanism that efficiently approximates high-order derivatives by sharing basis functions and incrementally accumulating low-rank components across orders, thereby avoiding redundant modeling. Under the condition that initial finite differences admit a linear decomposition, they theoretically prove that the matrix rank of high-order derivatives is monotonically non-increasing and that arbitrary rank sequences can be constructed. Integrating ordinary differential equation modeling, dynamic rank analysis, the generalized Leibniz rule, and a low-rank neural architecture, the method substantially reduces both parameter count and computational complexity while accurately capturing high-order dynamical features such as acceleration and jerk.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Personalized content ranking and presentation

Abstract

Diffusion models have become the de facto standard for modern visual generation, including well-established frameworks such as latent diffusion and flow matching. Recently, modeling high-order dynamics has emerged as a promising frontier in generative modeling. Rather than only learning the first-order velocity field that transports random noise to a target data distribution, these approaches simultaneously learn higher-order derivatives, such as acceleration and jerk, yielding a diverse family of higher-order diffusion variants. To represent higher-order derivatives, naive approaches instantiate separate neural networks for each order, which scales the parameter space linearly with the derivative order. To overcome this computational bottleneck, we introduce cascading low-rank fitting, an ordinary differential equation inspired method that approximates successive derivatives by applying a shared base function augmented with sequentially accumulated low-rank components. Theoretically, we analyze the rank dynamics of these successive matrix differences. We prove that if the initial difference is linearly decomposable, the generic ranks of high-order derivatives are guaranteed to be monotonically non-increasing. Conversely, we demonstrate that without this structural assumption, the General Leibniz Rule allows ranks to strictly increase. Furthermore, we establish that under specific conditions, the sequence of derivative ranks can be designed to form any arbitrary permutation. Finally, we present a straightforward algorithm to efficiently compute the proposed cascading low-rank fitting.

Problem

• high-order diffusion
• parameter scaling
• computational bottleneck
• derivative modeling
• generative modeling

Innovation

• cascading low-rank fitting
• high-order diffusion
• rank dynamics
• generative modeling
• ordinary differential equation