StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation

2025-05-10

AI Summary

This paper addresses single-image motion estimation—specifically for image stitching rectification and rolling-shutter correction—by proposing a geometry-content prior transfer method leveraging a pre-trained text-to-image diffusion model (Stable Diffusion). Methodologically, it (1) reparameterizes the diffusion model as a mapping from latent space to motion fields; (2) introduces an Adaptive Ensemble Strategy (AES) to enhance consistency across multiple sampling trajectories; and (3) identifies and exploits the “Sampling-Step Disaster” (SSD) phenomenon—wherein motion-field accuracy peaks at a single denoising step—enabling one-step, high-fidelity prediction and accelerating inference by 200×. The approach achieves state-of-the-art performance on both tasks, demonstrating strong generalization, sub-pixel accuracy, and real-time capability without task-specific architectural modifications or fine-tuning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present StableMotion, a novel framework leverages knowledge (geometry and content priors) from pretrained large-scale image diffusion models to perform motion estimation, solving single-image-based image rectification tasks such as Stitched Image Rectangling (SIR) and Rolling Shutter Correction (RSC). Specifically, StableMotion framework takes text-to-image Stable Diffusion (SD) models as backbone and repurposes it into an image-to-motion estimator. To mitigate inconsistent output produced by diffusion models, we propose Adaptive Ensemble Strategy (AES) that consolidates multiple outputs into a cohesive, high-fidelity result. Additionally, we present the concept of Sampling Steps Disaster (SSD), the counterintuitive scenario where increasing the number of sampling steps can lead to poorer outcomes, which enables our framework to achieve one-step inference. StableMotion is verified on two image rectification tasks and delivers state-of-the-art performance in both, as well as showing strong generalizability. Supported by SSD, StableMotion offers a speedup of 200 times compared to previous diffusion model-based methods.

Problem

• Repurposing diffusion models for motion estimation tasks
• Solving single-image rectification like stitching and rolling shutter
• Achieving fast inference with adaptive ensemble strategies

Innovation

• Repurposes Stable Diffusion for motion estimation
• Uses Adaptive Ensemble Strategy for consistency
• Achieves one-step inference via Sampling Steps Disaster