Master Rules from Chaos: Learning to Reason, Plan, and Interact from Chaos for Tangram Assembly

2025-05-17

AI Summary

This work addresses the challenges of lacking geometric/kinematic priors and poor target generalization in robotic tangram assembly. We propose the first end-to-end vision-driven assembly framework that requires no geometric modeling, no pretraining on target objects, and no handcrafted reward design. Our method employs self-exploratory reinforcement learning in simulation, where sparse rewards are derived solely from visual observation changes; it further integrates representation learning and policy transfer to enable zero-shot generalization across unseen tangram configurations. Experiments demonstrate high robustness and assembly success rates on entirely novel tangram compositions, and successful transfer to new tasks such as utensil pairing. These results validate the feasibility of strong generalization in assembly using only contour-based visual cues, establishing a new paradigm for generalizable, embodiment-aware manipulation learning.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Visual Reasoning & Symbolic Representations
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Tangram assembly, the art of human intelligence and manipulation dexterity, is a new challenge for robotics and reveals the limitations of state-of-the-arts. Here, we describe our initial exploration and highlight key problems in reasoning, planning, and manipulation for robotic tangram assembly. We present MRChaos (Master Rules from Chaos), a robust and general solution for learning assembly policies that can generalize to novel objects. In contrast to conventional methods based on prior geometric and kinematic models, MRChaos learns to assemble randomly generated objects through self-exploration in simulation without prior experience in assembling target objects. The reward signal is obtained from the visual observation change without manually designed models or annotations. MRChaos retains its robustness in assembling various novel tangram objects that have never been encountered during training, with only silhouette prompts. We show the potential of MRChaos in wider applications such as cutlery combinations. The presented work indicates that radical generalization in robotic assembly can be achieved by learning in much simpler domains.

Problem

• Robotic tangram assembly challenges reasoning, planning, and manipulation.
• Learning assembly policies for novel objects without prior models.
• Achieving generalization in assembly via self-exploration and visual feedback.

Innovation

• Learns assembly policies through self-exploration
• Uses visual observation change as reward signal
• Generalizes to novel objects with silhouette prompts