Learning 3D Texture-Aware Representations for Parsing Diverse Human Clothing and Body Parts

2025-08-08

AI Summary

Existing methods for human parsing suffer from two key limitations: (1) fixed label vocabularies hinder fine-grained clothing and body-part recognition, and (2) open-vocabulary segmentation often treats the human body as a single category, lacking pixel-level semantic discrimination and instance-aware decomposition. To address these challenges, we propose a texture-aware representation learning framework that pioneers the integration of image-driven 3D texture diffusion models (I2Tx) into human parsing. Leveraging prompt-guided generation and feature distillation, our method extracts structural-aligned, semantically sensitive 3D texture-aware features, enabling multi-instance disentanglement and open-vocabulary pixel-level semantic segmentation. Extensive cross-dataset experiments demonstrate significant improvements in zero-shot parsing accuracy and generalization—particularly for unseen clothing categories and full-body parts. Our approach establishes a novel paradigm for fine-grained, open-vocabulary human parsing with strong compositional and semantic expressivity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Existing methods for human parsing into body parts and clothing often use fixed mask categories with broad labels that obscure fine-grained clothing types. Recent open-vocabulary segmentation approaches leverage pretrained text-to-image (T2I) diffusion model features for strong zero-shot transfer, but typically group entire humans into a single person category, failing to distinguish diverse clothing or detailed body parts. To address this, we propose Spectrum, a unified network for part-level pixel parsing (body parts and clothing) and instance-level grouping. While diffusion-based open-vocabulary models generalize well across tasks, their internal representations are not specialized for detailed human parsing. We observe that, unlike diffusion models with broad representations, image-driven 3D texture generators maintain faithful correspondence to input images, enabling stronger representations for parsing diverse clothing and body parts. Spectrum introduces a novel repurposing of an Image-to-Texture (I2Tx) diffusion model -- obtained by fine-tuning a T2I model on 3D human texture maps -- for improved alignment with body parts and clothing. From an input image, we extract human-part internal features via the I2Tx diffusion model and generate semantically valid masks aligned to diverse clothing categories through prompt-guided grounding. Once trained, Spectrum produces semantic segmentation maps for every visible body part and clothing category, ignoring standalone garments or irrelevant objects, for any number of humans in the scene. We conduct extensive cross-dataset experiments -- separately assessing body parts, clothing parts, unseen clothing categories, and full-body masks -- and demonstrate that Spectrum consistently outperforms baseline methods in prompt-based segmentation.

Problem

• Parsing diverse human clothing and body parts accurately
• Overcoming limitations of fixed mask categories in human parsing
• Improving alignment with body parts using 3D texture generators

Innovation

• Repurposing Image-to-Texture diffusion model
• Fine-tuning T2I model on 3D texture maps
• Prompt-guided grounding for semantic masks