FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

2026-07-06

AI Summary

This work addresses the challenge of limited functional generalization in robotic manipulation when encountering novel tools. To overcome this, the authors propose FORGE, a two-stage approach that first predicts generalizable 2D keypoint trajectories from action-free data to capture functional intent, then maps these trajectories to executable robot actions using only a few demonstrations, thereby decoupling functional reasoning from motor execution. FORGE is the first method to demonstrate that keypoint trajectories effectively balance functional expressiveness and motor feasibility. Evaluated on a benchmark involving seven distinct striking tools, the method achieves more than a twofold average improvement in success rate over existing approaches on unseen tools, both in simulation and real-world settings.

Abstract

While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.

Problem

• functional generalization
• tool-use
• robotic manipulation
• action transfer
• affordance

Innovation

• functional generalization
• keypoint trajectory
• tool-use
• robotic policy
• affordance reasoning