Problem
Research questions and friction points this paper is trying to address.
geometry estimation
pretrained models
data efficiency
joint modeling
next-frames prediction
Video Generative Models as Geometry Learner
🤖 AI Summary
本文提出GeoNeXt，利用预训练视频生成模型进行几何估计，通过预测下一帧任务实现数据高效且有效的几何学习，优于现有方法。
📝 Abstract
Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.
Innovation
Methods, ideas, or system contributions that make the work stand out.
video generative models
geometry estimation
next-frames prediction
data-efficient
joint modeling
🔎 Similar Papers
No similar papers found.
