Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT

2026-05-10

AI Summary

This work proposes a lightweight knowledge distillation framework to transfer the 3D spatial reasoning capabilities of a large 7B-parameter teacher model (e.g., LLaVA-3D) to a compact 2.29B-parameter student model, addressing the high computational cost that hinders deployment. The key innovation lies in introducing, for the first time in 3D vision-language model distillation, a learnable implicit Chain-of-Thought (Hidden CoT) module as an internal reasoning scratchpad, coupled with a VGGT visual encoder and an uncertainty-aware multi-task loss weighting mechanism. Notably, the method significantly enhances reasoning performance without requiring explicit CoT annotations. The resulting student model achieves 68–72% accuracy on proximity and contact tasks over ScanNet and 3D-FRONT benchmarks—retaining 54–72% of the teacher’s performance—while reducing model size by approximately 3× and inference latency by 8.7×.

Abstract

Large-scale 3D vision-language models (VLMs) like LLaVA-3D offer strong spatial reasoning but are difficult to deploy due to high computational costs. We propose a knowledge distillation framework that transfers spatial reasoning from a 7B teacher to a 2.29B student model. Our approach achieves 8.7x lower inference latency and a 3x reduction in model size while retaining 54-72% of the teacher's performance. The framework utilizes VGGT as the vision encoder and a multi-task distillation pipeline with uncertainty-aware loss weighting. To improve reasoning without chain-of-thought (CoT) data, we introduce "Hidden CoT": learnable latent tokens that serve as an internal scratchpad before answer generation. This is the first use of latent scratchpad reasoning in distilled 3D VLMs. The student model jointly performs spatial description, depth estimation, and object detection. Experiments on ScanNet and 3D-FRONT show strong spatial understanding, reaching 68-72% accuracy on proximity and contact tasks. Our framework enables efficient 3D scene QA on resource-constrained platforms.

Problem

• 3D vision-language models
• spatial reasoning
• model compression
• knowledge distillation
• efficient deployment

Innovation

• knowledge distillation
• 3D vision-language model
• Hidden CoT
• latent scratchpad
• uncertainty-aware loss