ClimbingCap: Multi-Modal Dataset and Method for Rock Climbing in World Coordinate

2025-03-27

AI Summary

Existing human motion reconstruction (HMR) methods are primarily designed for ground-level locomotion and struggle to accurately recover global 3D pose during off-ground climbing activities—largely due to the absence of large-scale, climbing-specific datasets with 3D global coordinate annotations and HMR frameworks tailored to aerial, non-planar environments. To address this, we introduce AscendMotion, the first large-scale multimodal (RGB/LiDAR/IMU) climbing motion dataset, comprising 412k frames captured from 22 climbers traversing challenging routes on 12 distinct climbing walls. We further propose ClimbingCap, a novel end-to-end neural motion reconstruction framework featuring joint optimization across RGB and LiDAR coordinate systems, incorporating LiDAR-derived global geometric constraints to enforce spatial consistency. Extensive experiments demonstrate that ClimbingCap significantly improves global position and orientation accuracy over state-of-the-art HMR methods, establishing a new benchmark for off-ground human motion analysis.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Human Motion Recovery (HMR) research mainly focuses on ground-based motions such as running. The study on capturing climbing motion, an off-ground motion, is sparse. This is partly due to the limited availability of climbing motion datasets, especially large-scale and challenging 3D labeled datasets. To address the insufficiency of climbing motion datasets, we collect AscendMotion, a large-scale well-annotated, and challenging climbing motion dataset. It consists of 412k RGB, LiDAR frames, and IMU measurements, including the challenging climbing motions of 22 skilled climbing coaches across 12 different rock walls. Capturing the climbing motions is challenging as it requires precise recovery of not only the complex pose but also the global position of climbers. Although multiple global HMR methods have been proposed, they cannot faithfully capture climbing motions. To address the limitations of HMR methods for climbing, we propose ClimbingCap, a motion recovery method that reconstructs continuous 3D human climbing motion in a global coordinate system. One key insight is to use the RGB and LiDAR modalities to separately reconstruct motions in camera coordinates and global coordinates and to optimize them jointly. We demonstrate the quality of the AscendMotion dataset and present promising results from ClimbingCap. The AscendMotion dataset and source code release publicly at href{this link}{http://www.lidarhumanmotion.net/climbingcap/}

Problem

• Lack of large-scale 3D labeled climbing motion datasets
• Inability of existing HMR methods to capture climbing motions accurately
• Need for precise recovery of climber pose and global position

Innovation

• Uses RGB and LiDAR for motion reconstruction
• Joint optimization in camera and global coordinates
• Large-scale annotated climbing dataset AscendMotion