TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

2026-07-12

AI Summary

This work addresses the challenge of transferring knowledge from vision foundation models to LiDAR backbones, where existing cross-modal distillation methods struggle due to the simultaneous presence of modality and architectural discrepancies. To overcome this, the authors propose a decoupling strategy that introduces a student Vision Transformer (ViT) isomorphic to the teacher visual ViT. Point clouds are mapped to image-patch-aligned tokens via Frustum Pooling and Frustum Attention, enabling token-level distillation augmented with visibility-aware masks. Subsequently, masked bilinear sampling reconstructs point-level features from these tokens, facilitating deployment with LiDAR-only inputs. This approach is the first to disentangle modality adaptation from architectural differences, achieving substantial performance gains across five LiDAR datasets and four cross-sensor tasks, while remaining compatible with frozen backbones and lightweight prediction heads.

Abstract

Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for LiDAR representation learning that addresses this gap by coupling a LiDAR backbone with a student Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-token representations. TOLiD converts the set of point features within each image patch frustum into a token using Frustum Pooling followed by Frustum Attention, and performs token-level distillation with visibility masking. For LiDAR-only deployment, we lift token features back to per-point representations using masked bilinear sampling to avoid patches that have limited LiDAR points. We extensively evaluate TOLiD on five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads.

Problem

• cross-modal distillation
• Vision Foundation Models
• LiDAR pretraining
• architecture gap
• 3D scene understanding

Innovation

• cross-modal distillation
• token lifting
• frustum attention
• self-supervised pretraining
• vision foundation model