Learning to Manipulate Anything: Revealing Data Scaling Laws in Bounding-Box Guided Policies

2026-02-12

AI Summary

This work addresses the limited generalization of existing diffusion-based policies in semantic manipulation tasks, which struggle to accurately localize target objects in complex environments based on textual instructions. The authors propose a bounding box–guided diffusion policy that integrates object detection with a diffusion model through a semantics-to-motion decoupling framework, leveraging the Label-UMI automated annotation system to efficiently construct a semantically labeled demonstration dataset. They uncover, for the first time, a power-law relationship between generalization performance and the number of objects annotated with bounding boxes, enabling a data-efficient collection strategy. In large-scale real-world experiments, the method achieves an 85% success rate across four task categories on both seen and unseen objects, significantly enhancing generalization and scalability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Manipulation
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Diffusion-based policies show limited generalization in semantic manipulation, posing a key obstacle to the deployment of real-world robots. This limitation arises because relying solely on text instructions is inadequate to direct the policy's attention toward the target object in complex and dynamic environments. To solve this problem, we propose leveraging bounding-box instruction to directly specify target object, and further investigate whether data scaling laws exist in semantic manipulation tasks. Specifically, we design a handheld segmentation device with an automated annotation pipeline, Label-UMI, which enables the efficient collection of demonstration data with semantic labels. We further propose a semantic-motion-decoupled framework that integrates object detection and bounding-box guided diffusion policy to improve generalization and adaptability in semantic manipulation. Throughout extensive real-world experiments on large-scale datasets, we validate the effectiveness of the approach, and reveal a power-law relationship between generalization performance and the number of bounding-box objects. Finally, we summarize an effective data collection strategy for semantic manipulation, which can achieve 85\% success rates across four tasks on both seen and unseen objects. All datasets and code will be released to the community.

Problem

• semantic manipulation
• diffusion policy
• generalization
• bounding-box instruction
• data scaling laws

Innovation

• bounding-box guided policy
• data scaling laws
• semantic manipulation
• diffusion policy
• Label-UMI