Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning

2024-11-29

arXiv.org

AI Summary

Vision Transformers (ViTs) exhibit limited capability in understanding 3D spatial relationships, hindering their performance on geometrically grounded downstream tasks. Method: We propose a lightweight, architecture-agnostic multi-view equivariance enhancement method that (1) enforces geometric consistency via 3D correspondence-based constraints; (2) employs a contrastive fine-tuning strategy achieving significant improvement in cross-view equivariance of 3D semantic embeddings within a single object and a single training iteration; and (3) introduces equivariance regularization in feature space. Contribution/Results: We are the first to systematically demonstrate a strong correlation between multi-view semantic embedding equivariance and performance on pose estimation, tracking, and cross-view semantic transfer. Experiments show substantial gains across diverse 3D perception tasks—without requiring large-scale annotations or model architectural modifications.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vision foundation models, particularly the ViT family, have revolutionized image understanding by providing rich semantic features. However, despite their success in 2D comprehension, their abilities on grasping 3D spatial relationships are still unclear. In this work, we evaluate and enhance the 3D awareness of ViT-based models. We begin by systematically assessing their ability to learn 3D equivariant features, specifically examining the consistency of semantic embeddings across different viewpoints. Our findings indicate that improved 3D equivariance leads to better performance on various downstream tasks, including pose estimation, tracking, and semantic transfer. Building on this insight, we propose a simple yet effective finetuning strategy based on 3D correspondences, which significantly enhances the 3D correspondence understanding of existing vision models. Remarkably, finetuning on a single object for one iteration results in substantial gains. Our code is available at https://github.com/qq456cvb/3DCorrEnhance.

Problem

• Enhancing 3D awareness in ViT models
• Improving 3D equivariant feature learning
• Minimal finetuning for better 3D correspondence

Innovation

• Enhance 3D awareness via ViT
• Finetune with 3D correspondence strategy
• Improve 3D equivariance for better performance