Learning to Refocus with Video Diffusion Models

2025-12-22

AI Summary

Autofocus in photography frequently fails, and existing methods struggle to achieve photorealistic, controllable post-capture refocusing from a single defocused image. To address this, we propose the first end-to-end video diffusion-based framework for focal stack synthesis, enabling interactive, real-time refocusing and editing. Our key contributions are: (1) the first application of video diffusion models to perceptually realistic focal stack generation; (2) the construction of the first large-scale, real-world smartphone-captured focal stack dataset; and (3) a novel training paradigm integrating defocus modeling, real-domain image synthesis, and multi-scale spatiotemporal denoising. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches in perceptual quality, robustness to diverse defocus patterns, and generalization to complex scenes. Both code and dataset are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Focus is a cornerstone of photography, yet autofocus systems often fail to capture the intended subject, and users frequently wish to adjust focus after capture. We introduce a novel method for realistic post-capture refocusing using video diffusion models. From a single defocused image, our approach generates a perceptually accurate focal stack, represented as a video sequence, enabling interactive refocusing and unlocking a range of downstream applications. We release a large-scale focal stack dataset acquired under diverse real-world smartphone conditions to support this work and future research. Our method consistently outperforms existing approaches in both perceptual quality and robustness across challenging scenarios, paving the way for more advanced focus-editing capabilities in everyday photography. Code and data are available at www.learn2refocus.github.io

Problem

• Generates focal stacks from single defocused images
• Enables interactive post-capture refocusing in photography
• Outperforms existing methods in perceptual quality and robustness

Innovation

• Uses video diffusion models for refocusing
• Generates focal stack from single defocused image
• Enables interactive post-capture focus adjustment