GROW$^2$: Grounding Which and Where for Robot Tool Use

2026-06-29

AI Summary

This work addresses the problem of affordance grounding for robotic tool use in open-world settings—specifically, selecting appropriate tools from arbitrary object categories and precisely localizing their functional regions. The authors propose a hierarchical grounding framework that treats object parts as abstract units. This approach leverages vision-language models for task parsing, tool selection, and part identification, and integrates foundation vision models to accurately map identified parts to 3D manipulation regions, all from a single RGB-D image. Without requiring large-scale end-to-end training, the framework outperforms existing methods on standard affordance prediction benchmarks and demonstrates zero-shot generalization to open-category tools in both simulated and real-world robotic experiments.

Abstract

Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of $\textit{open-world affordance grounding}$: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW$^2$ (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW$^2$ harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.

Problem

• tool use
• affordance grounding
• open-world
• robot manipulation
• functional reasoning

Innovation

• affordance grounding
• tool use
• vision-language models
• zero-shot generalization
• part-based reasoning