Tri-Efficient Transfer Learning for Point Cloud Videos

2026-06-23

AI Summary

This work addresses the dual challenges of high annotation costs and memory-constrained fine-tuning in point cloud video understanding by proposing a unified transfer learning framework that simultaneously optimizes data, parameter, and memory efficiency. The framework introduces, for the first time, a tripartite efficiency co-optimization mechanism, integrating a geometry-motion dual network with a lightweight spatiotemporal side network to enable efficient fine-tuning while keeping the backbone frozen. Key technical components include pseudo motion trajectory synthesis, multimodal contrastive learning, rigid rotation prediction, LoRA adaptation modules, and a gradient flow masking strategy. Evaluated on action recognition and semantic segmentation tasks, the method achieves state-of-the-art performance while substantially reducing the number of trainable parameters and memory consumption.

Abstract

While point cloud foundation models have significantly advanced point cloud video understanding, existing parameter-efficient fine-tuning (PEFT) methods still suffer from two critical limitations: prohibitive annotation costs for large-scale point cloud datasets and severe memory bottlenecks. In this paper, we aim to mine richer supervision signals from existing data rather than blindly scaling datasets. A further key principle is that the memory footprint of fine-tuning must be drastically reduced compared to full fine-tuning, which remains elusive for current PEFT techniques. Driven by these challenges, we identify three core desiderata: data-, parameter-, and memory efficiency, and present PoinTriE, a unified framework that excels along all three dimensions. For pre-training, pseudo-motion trajectories are synthesized via rigid transformations, paired with text corpora and 2D projections derived from raw point clouds. We then propose a Geometric-Motion Duality Network optimized via multimodal contrastive learning, rigid rotation prediction, and motion distribution divergence to produce dense self-supervision. During fine-tuning, we freeze the pretrained backbone and only update a lightweight Spatio-temporal Side Network built with LoRA units. Equipped with a gradient flow masking strategy, PoinTriE simultaneously reduces memory consumption and parameter overhead. Extensive experiments confirm that PoinTriE establishes new state-of-the-art results on action recognition and semantic segmentation tasks.

Problem

• point cloud videos
• annotation cost
• memory bottleneck
• parameter-efficient fine-tuning
• data efficiency

Innovation

• parameter-efficient fine-tuning
• self-supervised learning
• point cloud video
• multimodal contrastive learning
• LoRA