POMA-3D: The Point Map Way to 3D Scene Understanding

2025-11-20

AI Summary

To address the scarcity of pretraining priors and limited labeled data in 3D representation learning, this paper introduces POMA-3D—the first self-supervised 3D representation model based on point-images. Methodologically, it structurally encodes unordered point clouds into 2D grid representations that preserve global geometry, explicitly embedding 3D coordinates without requiring RGB input; proposes a multi-view geometrically consistent joint embedding-prediction architecture (POMA-JEPA) with view-to-scene alignment; and constructs the large-scale ScenePoint pretraining dataset. Evaluated under pure 3D coordinate input, POMA-3D achieves significant performance gains across diverse downstream tasks—including 3D visual question answering, embodied navigation, scene retrieval, and localization—demonstrating strong generalization while maintaining task-specific efficacy. As a result, it establishes a powerful, versatile multi-task 3D backbone model.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordinates on a structured 2D grid, preserving global 3D geometry while remaining compatible with the input format of 2D foundation models. To transfer rich 2D priors into POMA-3D, a view-to-scene alignment strategy is designed. Moreover, as point maps are view-dependent with respect to a canonical space, we introduce POMA-JEPA, a joint embedding-predictive architecture that enforces geometrically consistent point map features across multiple views. Additionally, we introduce ScenePoint, a point map dataset constructed from 6.5K room-level RGB-D scenes and 1M 2D image scenes to facilitate large-scale POMA-3D pretraining. Experiments show that POMA-3D serves as a strong backbone for both specialist and generalist 3D understanding. It benefits diverse tasks, including 3D question answering, embodied navigation, scene retrieval, and embodied localization, all achieved using only geometric inputs (i.e., 3D coordinates). Overall, our POMA-3D explores a point map way to 3D scene understanding, addressing the scarcity of pretrained priors and limited data in 3D representation learning. Project Page: https://matchlab-imperial.github.io/poma3d/

Problem

• Developing self-supervised 3D representation learning from point maps
• Transferring rich 2D priors into 3D understanding through view alignment
• Addressing scarcity of pretrained models and data in 3D learning

Innovation

• Self-supervised 3D representation model from point maps
• View-to-scene alignment strategy transfers 2D priors
• Joint embedding-predictive architecture ensures geometric consistency