Token Adaptation via Side Graph Convolution for Temporally and Spatially Efficient Fine-tuning of 3D Point Cloud Transformers

2025-02-19

AI Summary

To address the high computational overhead and weak adaptation capability in parameter-efficient fine-tuning (PEFT) of 3D point cloud Transformers, this paper proposes STAG—a token dynamic adaptation method leveraging parallel side graph convolutional networks. Its core innovations include: (i) the first point-cloud-specific side graph convolutional token adaptation mechanism; (ii) gradient-thinned connections; (iii) cross-layer parameter sharing; and (iv) lightweight graph convolutions. For systematic evaluation, we introduce PCC13—the first PEFT-dedicated point cloud benchmark, comprising 13 diverse datasets. Experiments demonstrate that STAG achieves classification accuracy on par with full-parameter fine-tuning while requiring only 0.43M trainable parameters. It reduces GPU memory consumption by 62% and training time by 57%. Moreover, STAG consistently outperforms existing PEFT methods across multiple tasks in PCC13, establishing new state-of-the-art performance for efficient point cloud Transformer adaptation.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Parameter-efficient fine-tuning (PEFT) of pre-trained 3D point cloud Transformers has emerged as a promising technique for 3D point cloud analysis. While existing PEFT methods attempt to minimize the number of tunable parameters, they still suffer from high temporal and spatial computational costs during fine-tuning. This paper proposes a novel PEFT algorithm for 3D point cloud Transformers, called Side Token Adaptation on a neighborhood Graph (STAG), to achieve superior temporal and spatial efficiency. STAG employs a graph convolutional side network that operates in parallel with a frozen backbone Transformer to adapt tokens to downstream tasks. STAG's side network realizes high efficiency through three key components: connection with the backbone that enables reduced gradient computation, parameter sharing framework, and efficient graph convolution. Furthermore, we present Point Cloud Classification 13 (PCC13), a new benchmark comprising diverse publicly available 3D point cloud datasets, enabling comprehensive evaluation of PEFT methods. Extensive experiments using multiple pre-trained models and PCC13 demonstrates the effectiveness of STAG. Specifically, STAG maintains classification accuracy comparable to existing methods while reducing tunable parameters to only 0.43M and achieving significant reductions in both computational time and memory consumption for fine-tuning. Code and benchmark will be available at: https://github.com/takahikof/STAG

Problem

• Efficient fine-tuning of 3D point cloud Transformers
• Reducing computational and memory costs in fine-tuning
• Introducing a new benchmark for PEFT evaluation

Innovation

• Side Token Adaptation on a neighborhood Graph (STAG)
• Graph convolutional side network
• Parameter sharing framework