VLM6D: VLM based 6Dof Pose Estimation based on RGB-D Images

2025-10-31

AI Summary

Poor generalization of 6D object pose estimation in real-world scenarios—caused by illumination variations, textureless surfaces, and severe occlusion—remains a critical challenge. To address this, we propose a robust RGB-D dual-stream architecture. Our method pioneers the synergistic integration of the DINOv2 vision transformer for RGB feature extraction and PointNet++ for point cloud encoding, enabling effective cross-modal feature fusion. We further introduce a multi-task prediction head and a self-supervised auxiliary training strategy to jointly model geometric and semantic cues. Evaluated on the Occluded-LINEMOD dataset, our approach achieves state-of-the-art performance, improving ADD-S accuracy by 3.2% and significantly enhancing occlusion robustness. The framework delivers a generalizable solution for 6D pose estimation of textureless objects under complex lighting and heavy occlusion.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The primary challenge in computer vision is precisely calculating the pose of 6D objects, however many current approaches are still fragile and have trouble generalizing from synthetic data to real-world situations with fluctuating lighting, textureless objects, and significant occlusions. To address these limitations, VLM6D, a novel dual-stream architecture that leverages the distinct strengths of visual and geometric data from RGB-D input for robust and precise pose estimation. Our framework uniquely integrates two specialized encoders: a powerful, self-supervised Vision Transformer (DINOv2) processes the RGB modality, harnessing its rich, pre-trained understanding of visual grammar to achieve remarkable resilience against texture and lighting variations. Concurrently, a PointNet++ encoder processes the 3D point cloud derived from depth data, enabling robust geometric reasoning that excels even with the sparse, fragmented data typical of severe occlusion. These complementary feature streams are effectively fused to inform a multi task prediction head. We demonstrate through comprehensive experiments that VLM6D obtained new SOTA performance on the challenging Occluded-LineMOD, validating its superior robustness and accuracy.

Problem

• Estimating 6D object pose under varying lighting conditions
• Handling textureless objects and severe occlusions in pose estimation
• Generalizing from synthetic to real-world RGB-D data effectively

Innovation

• Dual-stream architecture combines RGB and depth data
• Self-supervised Vision Transformer processes RGB images
• PointNet++ encoder handles 3D point cloud data