Creative Robot Tool Use by Counterfactual Reasoning

2026-05-06

AI Summary

This work addresses the limited ability of robots to understand and transfer the causal properties of non-standard tools. It proposes the first framework integrating counterfactual reasoning with disentangled causal feature learning, leveraging dynamics simulation to uncover tool–task causal relationships. Guided by a vision-language model and augmented with geometric–physical perturbations, the approach generates novel tool representations that enable creative tool identification and skill generalization across objects. By jointly incorporating causal discovery, vision-language model feature extraction, counterfactual generation, and keypoint matching, the method significantly outperforms baseline approaches in tasks such as reaching, scooping, and high-reach retrieval, thereby enhancing both the reliability of tool selection and the accuracy of cross-task transfer.

Abstract

We propose a causal reasoning framework for creative robot tool use where a suitable tool for a task is correctly identified for use beyond its primary objectives. The proposed framework first discovers the causal relationships between the tool and the task by conducting simulated experiments in a dynamics model. We decouple the causal discovery problem into two complementary components: VLM-based feature suggestion and counterfactual tool generation via targeted geometric and physical feature perturbations. Then, novel objects are classified based on identified causal features, and the tool use skill is transferred via keypoint matching conditioned on the identified causal features. By reconstructing the task in a dynamics model, our approach grounds tool use in the physics of the problem. We illustrate our approach in reaching a distant object with different sticks, scooping candies from a bowl using diverse items, and using different boxes or crates as stepping platforms to retrieve an object from a high shelf. Our baseline comparisons show that identifying causal features and grounding them in physical tool properties leads to more reliable tool selection and stronger skill keypoint transfer.

Problem

• creative tool use
• counterfactual reasoning
• causal reasoning
• robotics
• tool selection

Innovation

• counterfactual reasoning
• causal discovery
• tool use
• dynamics model
• keypoint matching