VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining

2026-07-02

AI Summary

Existing feedforward 3D methods struggle to achieve effective geometry–semantics joint learning due to reliance on costly geometric supervision or incomplete photometric self-supervision. This work proposes Visual–Language Reprojection Consistency (VLRC) as a scalable unsupervised objective that leverages frozen vision–language models—such as CLIP—to provide cross-view semantic supervision, aligning monocular 3D reconstructions with language-anchored features without any 3D annotations. By enforcing reprojection-driven cross-view feature consistency, VLRC enables joint optimization of geometry and semantics under an open-vocabulary setting. Experiments demonstrate that VLRC significantly improves 3D reconstruction accuracy, depth estimation, and camera pose quality on both indoor and outdoor benchmarks, while substantially enhancing open-vocabulary 3D semantic segmentation performance in zero-shot settings.

Abstract

Feed-forward 3D models are commonly trained using either expensive geometric supervision or self-supervised photometric objectives, both of which provide incomplete learning signals. We introduce Vision-Language Reprojection Consistency (VLRC), a scalable auxiliary objective that exploits frozen vision-language representations as semantic multi-view supervision. Given a predicted 3D reconstruction, VLRC reprojects dense vision-language features across views and enforces feature consistency between corresponding image locations, requiring no additional 3D annotations. The objective integrates seamlessly with both self-supervised monocular reconstruction and supervised-pretrained feed-forward 3D models during unlabeled adaptation. By aligning geometry with language-grounded features, VLRC not only improves depth and camera estimation but also enables more coherent multi-view semantic fusion for open-vocabulary 3D scene understanding. Experiments on indoor and outdoor benchmarks demonstrate consistent gains in 3D reconstruction accuracy and zero-shot open-vocabulary 3D semantic segmentation.

Problem

• 3D pretraining
• incomplete learning signals
• geometric supervision
• photometric objectives
• open-vocabulary 3D scene understanding

Innovation

• Vision-Language Reprojection Consistency
• self-supervised 3D pretraining
• multi-view semantic supervision
• open-vocabulary 3D understanding
• feature consistency