Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots

2026-07-01

AI Summary

This work addresses the challenge of identity loss and motion discontinuity in multi-person 3D human mesh reconstruction from in-the-wild videos with frequent camera cuts. It presents the first cross-shot 3D human mesh tracking method capable of handling multi-person scenes. By constructing a unified 3D scene prior, the approach maps boundary frames from different shots into a shared 3D space and jointly optimizes cross-frame mesh registration and identity consistency to achieve temporally coherent and identity-preserving reconstructions. The method significantly outperforms state-of-the-art approaches in 3D human reconstruction, camera pose estimation, and identity tracking, enabling high-fidelity, continuous multi-person motion reconstruction across camera cuts for the first time.

Abstract

Tracking multi-person 3D human meshes from in-the-wild videos is a highly challenging problem due to complex interactions, frequent occlusions, and severe truncation inherent in unconstrained environments. While recent approaches have improved robustness against these issues, they largely overlook the critical challenge prevalent in real-world footage: frequent shot changes. These abrupt transitions in camera viewpoints often cause existing methods to lose track of human identities and fail in reconstructing temporally coherent trajectories. Although several recent works have explored 3D human mesh tracking under shot changes, they are still limited to single-person scenarios, making them inadequate for real-world videos where multiple people interact and appear simultaneously. To address this limitation, we propose Multi-THuMBS (Multi-person Tracking of 3D Human Meshes Beyond Video Shots) that leverages a state-of-the-art 3D scene prior to reconstruct the two boundary frames in a single shared 3D space. Human meshes are then registered within the shared 3D space, maintaining per-person identity and motion consistency across shot changes. Extensive experiments demonstrate that our approach yields significant improvements in 3D human mesh recovery, camera pose estimation, and identity tracking, thereby ensuring high-fidelity motion reconstruction with consistent identity preservation across shots compared to previous state-of-the-art methods.

Problem

• multi-person tracking
• 3D human mesh
• shot changes
• identity consistency
• in-the-wild videos

Innovation

• multi-person 3D tracking
• shot change handling
• 3D human mesh reconstruction
• identity consistency
• shared 3D scene prior