One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning

2026-09-07

AI Summary

本文提出SOfA模型，通过引入可学习的标准化关节槽和语义关节嵌入解决不同传感器间骨架数据异质性问题，实现跨传感器统一骨架表示学习。

Abstract

For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.

Problem

• Self-supervised learning
• skeleton data
• heterogeneity
• sensor-specific models
• generalizable motion representations

Innovation

• Self-supervised learning
• Canonical Joint Slots
• Semantic Joint Embedding
• Cross-sensor unification