AutoComPose: Automatic Generation of Pose Transition Descriptions for Composed Pose Retrieval Using Multimodal LLMs

2025-03-28

AI Summary

To address the scarcity, inconsistency, and high cost of pose-transition descriptions in Compositional Pose Retrieval (CPR), this paper introduces the first automatic pose-transition annotation framework based on multimodal large language models (MLLMs). Methodologically, it innovatively integrates body-part-level motion decomposition, mirror/exchange variant generation, and cycle-consistency constraints, coupled with structured prompt engineering to enhance logical coherence, fine-grainedness, and lexical diversity of generated descriptions. Based on this framework, we construct and publicly release two new CPR benchmarks: AIST-CPR and PoseFixCPR. Extensive experiments demonstrate that our automatically generated descriptions significantly outperform both human annotations and heuristic baselines in retrieval accuracy, achieving substantial performance gains. Moreover, the proposed framework reduces annotation costs by over 90%, enabling scalable, high-quality CPR dataset construction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Composed pose retrieval (CPR) enables users to search for human poses by specifying a reference pose and a transition description, but progress in this field is hindered by the scarcity and inconsistency of annotated pose transitions. Existing CPR datasets rely on costly human annotations or heuristic-based rule generation, both of which limit scalability and diversity. In this work, we introduce AutoComPose, the first framework that leverages multimodal large language models (MLLMs) to automatically generate rich and structured pose transition descriptions. Our method enhances annotation quality by structuring transitions into fine-grained body part movements and introducing mirrored/swapped variations, while a cyclic consistency constraint ensures logical coherence between forward and reverse transitions. To advance CPR research, we construct and release two dedicated benchmarks, AIST-CPR and PoseFixCPR, supplementing prior datasets with enhanced attributes. Extensive experiments demonstrate that training retrieval models with AutoComPose yields superior performance over human-annotated and heuristic-based methods, significantly reducing annotation costs while improving retrieval quality. Our work pioneers the automatic annotation of pose transitions, establishing a scalable foundation for future CPR research.

Problem

• Scarcity and inconsistency of annotated pose transitions hinder CPR progress
• Existing CPR datasets rely on costly human annotations or heuristic rules
• AutoComPose automates pose transition description generation using MLLMs

Innovation

• Uses multimodal LLMs for pose transition generation
• Structures transitions into fine-grained body movements
• Ensures logical coherence with cyclic consistency