DETRAM: End-to-end DEtection, Tracking and Recovery of HumAn Meshes

2026-07-10

AI Summary

This work addresses the challenges of multi-person 3D mesh recovery under occlusion and interaction scenarios, where joint detection, reconstruction, and cross-frame tracking remain difficult. The authors propose DETRAM, a unified framework that, for the first time, integrates promptability, multi-person 3D mesh recovery, and identity-consistent tracking within an end-to-end trainable Transformer architecture. By employing a shared decoder, DETRAM simultaneously performs detection, reconstruction, and tracking, leveraging learnable identity-consistent queries—including detection, tracking, and user-prompt queries—to enable user-guided tracking of specific individuals. Experiments demonstrate that DETRAM achieves state-of-the-art tracking performance on PoseTrack21, 3DPW, BEDLAM, and MuPoTS-3D, while also delivering competitive 3D reconstruction accuracy on BEDLAM and 3DPW.

Abstract

In the task of human mesh recovery (HMR), multi-person scenes are particularly difficult to handle due to the many entities that appear and occlusions between them over time. In particular for video inputs, there is a need to track each entity reliably and consistently. Existing methods rely on pretrained human detection modules, increasing their runtime and limiting the number of tracked entities. We present DETRAM, a unified framework for multi-person HMR and tracking that simultaneously detects, reconstructs, and tracks humans across time, both automatically and via user prompts. DETRAM uses a single transformer decoder with an identity-consistent set of learnable query embeddings that persist across frames: detection queries discover new people, tracking queries maintain pose and shape for existing individuals, and prompt queries follow user-specified identities. Our approach achieves state-of-the-art tracking results on PoseTrack21, 3DPW, BEDLAM, and MuPoTS-3D, and competitive reconstruction accuracy on BEDLAM and 3DPW, while uniquely supporting prompt-based tracking of individuals in multi-person scenes. To our knowledge, this is the first method to unify promptability and multi-person HMR with tracking in an end-to-end trainable framework, enabling user-directed human analysis in videos.

Problem

• human mesh recovery
• multi-person scenes
• occlusion
• video tracking
• identity consistency

Innovation

• end-to-end
• promptable tracking
• human mesh recovery
• identity-consistent queries
• multi-person video analysis