🤖 AI Summary
Existing human-centric perception models suffer from modality fragmentation and task-specific fine-tuning, limiting generalization across diverse downstream tasks. This paper introduces the first multimodal foundation model for human-centric tasks, unifying 2D/3D vision, skeletal action modeling, and vision-language understanding without task-specific adaptation. Our approach features three key innovations: (1) a novel dual universal head architecture that jointly models discrete representations (e.g., pose labels) and continuous ones (e.g., 3D coordinates); (2) a “cross-modal knowledge translation” paradigm that bridges semantic gaps via modality-aligned embeddings and unified sequence-based representation; and (3) multi-task joint pretraining coupled with prompt-driven inference. Evaluated on 12 benchmarks spanning eight task categories, our model achieves state-of-the-art performance on 11 tasks, demonstrating substantial improvements in cross-task generalization and transferability.
📝 Abstract
Human-centric perception tasks, e.g., pedestrian detection, skeleton-based action recognition, and pose estimation, have wide industrial applications, such as metaverse and sports analysis. There is a recent surge to develop human-centric foundation models that can benefit a broad range of human-centric perception tasks. While many human-centric foundation models have achieved success, they did not explore 3D and vision-language tasks for human-centric and required task-specific finetuning. These limitations restrict their application to more downstream tasks and situations. To tackle these problems, we present Hulk, the first multimodal human-centric generalist model, capable of addressing 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific finetuning. The key to achieving this is condensing various task-specific heads into two general heads, one for discrete representations, e.g., languages, and the other for continuous representations, e.g., location coordinates. The outputs of two heads can be further stacked into four distinct input and output modalities. This uniform representation enables Hulk to treat diverse human-centric tasks as modality translation, integrating knowledge across a wide range of tasks. Comprehensive evaluations of Hulk on 12 benchmarks covering 8 human-centric tasks demonstrate the superiority of our proposed method, achieving state-of-the-art performance in 11 benchmarks.
