Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning

2026-06-16

AI Summary

This work addresses the challenges of overfitting and catastrophic forgetting in few-shot domain adaptation for 3D vision-language foundation models by proposing ReFine3D, a novel framework that selectively fine-tunes model layers while enhancing cross-modal diversity through multi-view point cloud consistency constraints and synonymic textual prompts generated by large language models. The approach further incorporates visual supervision from rendered point clouds and a lightweight test-time confidence aggregation mechanism. Evaluated across multiple 3D domain generalization benchmarks, ReFine3D consistently outperforms state-of-the-art methods, achieving notable gains of 1.36% in base-to-novel category generalization, 2.43% in cross-dataset transfer, 1.80% in robustness against perturbations, and up to 3.11% in few-shot classification accuracy.

Abstract

Domain adaptation remains a central challenge in 3D vision, especially for multimodal foundation models that align 3D point clouds with visual and textual data. While these models demonstrate strong general capabilities, adapting them to downstream domains with limited data often leads to overfitting and catastrophic forgetting. To address this, we introduce ReFine3D, a regularized fine-tuning framework designed for domain-generalizable tuning of 3D large multimodal models (LMMs). ReFine3D combines selective layer tuning with two targeted regularization strategies: multi-view consistency across augmented point clouds and text diversity through synonym-based prompts generated by large language models. Additionally, we incorporate point-rendered vision supervision and a test-time augmentation mechanism with confidence-based aggregation to further enhance robustness. Extensive experiments across different 3D domain generalization benchmarks show that ReFine3D improves base-to-novel class generalization by 1.36%, cross-dataset transfer by 2.43%, robustness to corruption by 1.80%, and few-shot accuracy by up to 3.11%, outperforming prior state-of-the-art methods with minimal added computational overhead.

Problem

• domain adaptation
• 3D vision-language models
• catastrophic forgetting
• overfitting
• domain generalization

Innovation

• domain generalization
• 3D vision-language models
• regularized fine-tuning
• multi-view consistency
• text diversity