InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning

2025-05-23

AI Summary

Existing vision-language models (VLMs) treat objects as holistic entities, lacking explicit understanding of manipulable parts and their functional semantics—hindering task-oriented, fine-grained interaction. Method: We propose a task-driven part segmentation paradigm and introduce InstructPart, the first real-world, instruction-guided benchmark featuring multi-scene, human-annotated part masks and functionally grounded instructions. We formally define and evaluate part-level task understanding, and present a tri-modal instruction–image–part joint modeling framework that integrates instruction encoding with image-part alignment via multimodal feature learning, employing lightweight adapters for efficient fine-tuning. Contribution/Results: Our method achieves over 2× performance gain on InstructPart, revealing a critical capability gap in current VLMs for part-level reasoning. InstructPart establishes a new evaluation standard and technical foundation for robotics manipulation, VR interaction, and other embodied AI applications.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Manipulation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/.

Problem

• Enhancing object part segmentation for task-oriented applications
• Addressing limitations of current models in part-level understanding
• Improving Vision-Language Models' performance in real-world part segmentation tasks

Innovation

• Introduces InstructPart benchmark for part segmentation
• Fine-tunes VLMs for twofold performance improvement
• Enhances VLM applicability in robotics and VR