Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

2026-07-01

AI Summary

This work addresses the lack of effective cross-modal pretraining methods in self-supervised representation learning for dynamic 4D point clouds, which has largely been confined to single-modality approaches or global embeddings that overlook the rich local semantics encoded in pretrained 2D foundation models. We propose Cross4D-JEPA, the first framework to enable dense cross-modal knowledge distillation from frozen 2D/video foundation models—such as DINOv2 and V-JEPA 2—to a 4D point cloud encoder. Through a teacher-student architecture with point-wise feature matching, our method performs self-supervised pretraining without masking, negative sampling, or a decoder. By preserving patch-level semantics, Cross4D-JEPA yields significantly finer-grained and more transferable representations, outperforming existing single-modal and global cross-modal approaches across four benchmarks. Notably, it achieves performance on par with much larger backbones using an encoder 13× smaller, while also improving label efficiency and cross-domain generalization.

Abstract

Automatic understanding of dynamic 4D point clouds, the 3D-point sequences captured over time by depth sensors and LiDAR, is central to robotics and embodied perception. Yet annotating them densely is expensive, making self-supervised pretraining the natural route to transferable representations. Existing pretext tasks, however, are almost entirely intra-modal, and the few methods that transfer knowledge from 2D foundation models rely on a single global embedding per clip, discarding the rich per-patch semantics that these models compute. To address this gap, we propose Cross4D-JEPA, a teacher-student method that distills a frozen 2D foundation model, an image model DINOv2, or a video model V-JEPA 2, into a 4D point encoder. The proposed method combines (1) a dense cross-modal correspondence that maps every 3D point to the teacher patch feature it projects to, and (2) a per-point objective that trains the student to match these features in latent space with no masking, negatives, or decoder. We evaluate Cross4D-JEPA on four benchmarks, MSR-Action3D, DeformingThings4D, NTU-RGB+D 60, and HOI4D, against intra-modal and global cross-modal baselines. Experimental results show that, under a matched protocol, the proposed method consistently outperforms intra-modal and global cross-modal baselines across the four benchmarks and is competitive with heavier published 4D methods; further analysis attributes this gain primarily to the granularity of the correspondence rather than the teacher modality. Beyond recognition accuracy, the dense representation learned by Cross4D-JEPA transfers across domains, improves label efficiency, and improves full-label fine-tuning under the same training budget, while a 13x smaller encoder matches a heavyweight pooling backbone.

Problem

• 4D point cloud
• self-supervised learning
• cross-modal correspondence
• representation learning
• dense distillation

Innovation

• dense cross-modal correspondence
• 4D point cloud representation
• knowledge distillation
• self-supervised learning
• foundation model transfer