Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

2025-10-27

AI Summary

This work addresses weak semantic consistency and challenging cross-modal alignment in unsupervised 3D representation learning by proposing the first joint self-supervised framework for 2D images and 3D point clouds. Methodologically, it integrates 2D–3D cross-modal contrastive learning, intra-modal self-distillation on point clouds, video-guided temporal modeling, and linear projection into the CLIP language embedding space—mimicking human multimodal cognitive integration. Its core contribution is enabling geometry-semantic joint representation learning without manual annotations, while supporting open-world semantic transfer. On ScanNet, zero-shot linear probing achieves an mIoU of 80.7%, outperforming the best prior 2D and 3D baselines by 14.2% and 4.8%, respectively. Visualizations confirm strong alignment between learned geometric structures and semantic distributions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2% and 4.8%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP's language space, enabling open-world perception. These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency.

Problem

• Learning spatial representations through joint 2D-3D self-supervised learning
• Improving 3D scene perception via multimodal feature coherence
• Enabling open-world spatial understanding with geometric-semantic consistency

Innovation

• Joint 2D-3D cross-modal embedding for spatial learning
• 3D intra-modal self-distillation combined with joint embedding
• Linear projection into CLIP's language space for open-world perception