MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning

2025-03-24

AI Summary

Addressing the challenge of parameter-efficient fine-tuning (PEFT) for 3D point cloud representation learning, this paper proposes MoST—the first reparameterization-based PEFT framework specifically designed for point clouds. Its core innovation is the Point Monarch—a structured sparse matrix that introduces Monarch decomposition into PEFT for the first time, enabling sparse parameter updates with zero inference overhead. MoST integrates reparameterization, sparse weight decomposition, and hybrid low-rank/Kronecker decomposition into a composable architecture, preserving local geometric modeling capability while enhancing representational expressiveness. Evaluated on ScanObjectNN (PB_T50_RS) and ModelNet40, MoST achieves state-of-the-art accuracy of 97.5% and 96.2%, respectively—significantly outperforming existing 3D PEFT methods—and supports further parameter compression without performance degradation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce Monarch Sparse Tuning (MoST), the first reparameterization-based parameter-efficient fine-tuning (PEFT) method tailored for 3D representation learning. Unlike existing adapter-based and prompt-tuning 3D PEFT methods, MoST introduces no additional inference overhead and is compatible with many 3D representation learning backbones. At its core, we present a new family of structured matrices for 3D point clouds, Point Monarch, which can capture local geometric features of irregular points while offering high expressiveness. MoST reparameterizes the dense update weight matrices as our sparse Point Monarch matrices, significantly reducing parameters while retaining strong performance. Experiments on various backbones show that MoST is simple, effective, and highly generalizable. It captures local features in point clouds, achieving state-of-the-art results on multiple benchmarks, e.g., 97.5% acc. on ScanObjectNN (PB_50_RS) and 96.2% on ModelNet40 classification, while it can also combine with other matrix decompositions (e.g., Low-rank, Kronecker) to further reduce parameters.

Problem

• Efficient fine-tuning for 3D representation learning
• Reducing parameters without inference overhead
• Capturing local geometric features in point clouds

Innovation

• Reparameterization-based PEFT for 3D learning
• Point Monarch matrices capture local features
• Combines with decompositions to reduce parameters