MAMMA: Markerless&Automatic Multi-Person Motion Action Capture

2025-06-16

AI Summary

Traditional marker-based motion capture relies on specialized hardware and manual calibration, resulting in high costs and labor-intensive workflows; existing learning-based approaches are largely limited to single-subject, sparse 2D keypoints and struggle with occlusion and physical human-human interaction. To address these challenges, MAMMA introduces the first markerless, multi-view motion capture framework tailored for two-person interaction. Its core innovations include: (1) segmentation-mask-guided prediction of dense 2D surface keypoints; (2) a learnable query mechanism enabling robust identity association and correspondence estimation under severe occlusion; and (3) the first large-scale synthetic multi-view interactive motion capture dataset with SMPL-X ground truth. Evaluated on real-world two-person interaction sequences, MAMMA achieves SMPL-X parameter reconstruction accuracy comparable to commercial marker-based systems—without manual post-processing. The code, models, dataset, and benchmark are fully open-sourced.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present MAMMA, a markerless motion-capture pipeline that accurately recovers SMPL-X parameters from multi-view video of two-person interaction sequences. Traditional motion-capture systems rely on physical markers. Although they offer high accuracy, their requirements of specialized hardware, manual marker placement, and extensive post-processing make them costly and time-consuming. Recent learning-based methods attempt to overcome these limitations, but most are designed for single-person capture, rely on sparse keypoints, or struggle with occlusions and physical interactions. In this work, we introduce a method that predicts dense 2D surface landmarks conditioned on segmentation masks, enabling person-specific correspondence estimation even under heavy occlusion. We employ a novel architecture that exploits learnable queries for each landmark. We demonstrate that our approach can handle complex person--person interaction and offers greater accuracy than existing methods. To train our network, we construct a large, synthetic multi-view dataset combining human motions from diverse sources, including extreme poses, hand motions, and close interactions. Our dataset yields high-variability synthetic sequences with rich body contact and occlusion, and includes SMPL-X ground-truth annotations with dense 2D landmarks. The result is a system capable of capturing human motion without the need for markers. Our approach offers competitive reconstruction quality compared to commercial marker-based motion-capture solutions, without the extensive manual cleanup. Finally, we address the absence of common benchmarks for dense-landmark prediction and markerless motion capture by introducing two evaluation settings built from real multi-view sequences. We will release our dataset, benchmark, method, training code, and pre-trained model weights for research purposes.

Problem

• Accurately captures two-person interactions without markers
• Overcomes occlusion challenges in multi-person motion capture
• Eliminates need for manual cleanup in motion reconstruction

Innovation

• Markerless motion-capture using multi-view video
• Dense 2D landmarks from segmentation masks
• Learnable queries for landmark prediction