Cross-Attentive Multiview Fusion of Vision-Language Embeddings

2026-04-14

AI Summary

This work addresses the suboptimal 3D representation resulting from insufficient multi-view feature fusion by proposing a novel multi-view Transformer architecture. It introduces cross-attention mechanisms to effectively integrate multi-view vision-language embeddings, producing unified per-instance 3D semantic representations. Furthermore, it innovatively leverages multi-view consistency as a self-supervised signal to enhance representation learning. The proposed method significantly outperforms baseline approaches—such as average fusion or single-view strategies—on standard benchmarks for 3D semantic and instance classification. Notably, it also demonstrates strong generalization capabilities in zero-shot evaluations on out-of-domain datasets.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Vision-language models have been key to the development of open-vocabulary 2D semantic segmentation. Lifting these models from 2D images to 3D scenes, however, remains a challenging problem. Existing approaches typically back-project and average 2D descriptors across views, or heuristically select a single representative one, often resulting in suboptimal 3D representations. In this work, we introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints and fuses them into a unified per-3D-instance embedding. As a second contribution, we leverage multiview consistency as a self-supervision signal for this fusion, which significantly improves performance when added to a standard supervised target-class loss. Our Cross-Attentive Multiview Fusion, which we denote with its acronym CAMFusion, not only consistently outperforms naive averaging or single-view descriptor selection, but also achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets.

Problem

• vision-language models
• 3D semantic segmentation
• multiview fusion
• 3D scene understanding
• cross-view representation

Innovation

• cross-attention
• multiview fusion
• vision-language embedding
• self-supervision
• 3D semantic segmentation