SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens

2024-11-29

arXiv.org

AI Summary

This work addresses two key challenges in real-time multi-person 3D human mesh estimation from a single RGB image: excessive computational cost induced by high-resolution inputs and low estimation efficiency for small-scale individuals (e.g., distant persons). To this end, we propose a scale-adaptive token mechanism that dynamically allocates feature map resolution based on detection box size—preserving fine-grained representations in critical regions while distilling background features. Built upon the DETR architecture, our method integrates multi-scale feature encoding, dynamic token scheduling, and a lightweight decoder. It achieves state-of-the-art accuracy (comparable MPJPE) while significantly reducing computational complexity, enabling >30 FPS inference on a single GPU. The core innovation lies in the first-ever introduction of scale-adaptive tokens, explicitly coupling token resolution with the image-space scale of human instances—thereby jointly optimizing accuracy for small targets and efficiency for large ones.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We propose a one-stage framework for real-time multi-person 3D human mesh estimation from a single RGB image. While current one-stage methods, which follow a DETR-style pipeline, achieve state-of-the-art (SOTA) performance with high-resolution inputs, we observe that this particularly benefits the estimation of individuals in smaller scales of the image (e.g., those far from the camera), but at the cost of significantly increased computation overhead. To address this, we introduce scale-adaptive tokens that are dynamically adjusted based on the relative scale of each individual in the image within the DETR framework. Specifically, individuals in smaller scales are processed at higher resolutions, larger ones at lower resolutions, and background regions are further distilled. These scale-adaptive tokens more efficiently encode the image features, facilitating subsequent decoding to regress the human mesh, while allowing the model to allocate computational resources more effectively and focus on more challenging cases. Experiments show that our method preserves the accuracy benefits of high-resolution processing while substantially reducing computational cost, achieving real-time inference with performance comparable to SOTA methods.

Problem

• Real-time multi-person 3D mesh estimation from single RGB image
• Reduce computation overhead while maintaining high-resolution accuracy
• Dynamic scale-adaptive tokens for efficient feature encoding

Innovation

• Scale-adaptive tokens dynamically adjust resolution
• One-stage DETR framework for real-time mesh
• Efficient feature encoding reduces computational cost