Multi-HMR 2: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking

2026-06-12

AI Summary

This work addresses the limitations of existing human mesh reconstruction methods, which often rely on fixed camera assumptions or manual post-processing and thus struggle to achieve accurate metric 3D localization and multi-person tracking. The authors propose an end-to-end DETR-based framework that jointly performs multi-person detection, camera-centric mesh reconstruction, and cross-frame identity tracking—without requiring ground-truth camera parameters or video input. Their approach innovatively integrates SAM2 image memory features with unsupervised feature distillation to enforce identity consistency across frames and simultaneously predicts scene-consistent camera parameters. While maintaining state-of-the-art performance in pelvis-centered mesh reconstruction, the method significantly improves detection accuracy and metric 3D localization fidelity in the camera coordinate system.

Abstract

Most advances in human mesh recovery (HMR) have focused on pelvis-centered recovery, overlooking metric 3D localization and detection accuracy in the camera coordinate system - two key factors for real-world applications such as human-robot interaction and social scene understanding. Current evaluation protocols often ignore these aspects, emphasizing per-person, root-centered recovery rather than camera-space perception. As a result, existing approaches rely on fixed camera assumptions or handcrafted post-processing, limiting their robustness and practical deployment. We introduce Multi-HMR 2, a simple yet robust DETR-based framework for Multi-person Camera-centric Human detection, mesh Recovery, and tracking. Multi-HMR 2 predicts a scene-consistent camera together with human meshes, enabling metric 3D localization without ground-truth intrinsics. Moreover, by distilling image-based memory features from SAM2, Multi-HMR 2 extends to tracking, achieving consistent identity association without video supervision. Despite its conceptual simplicity - no handcrafted components, no video input, and no ground-truth cameras - Multi-HMR 2 achieves state-of-the-art pelvis-centered performance while substantially improving detection accuracy and metric 3D localization.

Problem

• human mesh recovery
• camera-centric localization
• multi-person detection
• metric 3D localization
• real-world applications

Innovation

• camera-centric human mesh recovery
• metric 3D localization
• DETR-based multi-person detection
• identity-consistent tracking
• SAM2 feature distillation