Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models

2026-06-19

AI Summary

This work addresses the challenge of robustly constructing lightweight, semantically consistent 3D representations from noisy multi-view 2D foundation model embeddings. It introduces Casper3D, a novel framework that, for the first time, fuses multi-view information through a lightweight probabilistic formulation: treating multi-view 2D semantic features as noisy observations of an underlying 3D semantic state and leveraging known camera relative poses, it employs ensemble variational Bayesian inference to achieve open-vocabulary 3D semantic understanding. By maintaining alignment with vision–language semantic spaces, the approach supports both language-guided and self-supervised embeddings. Extensive experiments demonstrate that Casper3D significantly outperforms naive multi-view pooling under noise and ambiguity, yielding more stable and generalizable 3D semantic representations.

Abstract

We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic representation. We model view-level semantic features as noisy observations of an underlying 3D semantic state and infer this state with a set-based variational model that incorporates relative pose during multi-view reasoning. Casper3D is trained by predicting held-out semantic observations from novel viewpoints, while remaining aligned with visual and text semantic spaces for open-vocabulary 3D understanding. The framework is backbone-agnostic and applies to both language-aligned and self-supervised embeddings. Experiments show that Casper3D produces more stable 3D semantics than simple multi-view pooling, especially in ambiguous and noisy settings.

Problem

• 3D semantic representation
• multi-view 2D embeddings
• noise robustness
• open-vocabulary 3D understanding
• lightweight pretraining

Innovation

• Bayesian inversion
• 3D semantic representation
• multi-view reasoning
• foundation models
• open-vocabulary 3D understanding