ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies

2026-06-15

AI Summary

Existing general-purpose manipulation policies struggle to reveal their proficiency in atomic skills and compositional generalization in real-world settings, often masking underlying deficiencies. This work proposes ATOM-Bench, the first benchmark that jointly evaluates atomic skill mastery and compositional generalization in physical environments by decomposing tabletop manipulation into atomic actions and instructions. The benchmark encompasses 30 atomic tasks and 24 unseen compositional tasks, evaluated through 2,700 physical rollouts on dual-arm and single-arm robotic platforms using 3,000 human demonstrations and novel metrics—Atomic Score and Compositional Failure Share—across five representative policies. Results demonstrate that while current approaches can align with simple instructions, they exhibit significant weaknesses in fine-grained manipulation, counting, and logical filtering, and strong performance on atomic tasks does not reliably transfer to compositional scenarios.

Abstract

Generalist manipulation policies are increasingly presented as foundation models for robotic control, but their real-world generalization remains difficult to diagnose. A policy may succeed on demonstrated tasks while still failing to execute fine-grained atomic skills or recombine learned skills in new task structures. We introduce \textbf{ATOM-Bench}, a real-world benchmark for evaluating both atomic skills and compositional generalization in manipulation policies. ATOM-Bench factorizes tabletop manipulation into motor atoms and instruction atoms, and contains 30 atomic tasks and 24 held-out compositional tasks across paired single-arm and dual-arm robot tracks. We collect 3,000 human demonstrations for atomic fine-tuning and release both the demonstration data and evaluation rollout data to support reproducible real-world evaluation. Policies are fine-tuned on atomic tasks and evaluated on both atomic skill acquisition and held-out compositional tasks. We further introduce Atomic Score (AS) and Compositional Failure Share (CFS) to distinguish failures caused by weak atomic skills from failures caused by limited compositional reuse. Through 2,700 physical rollouts on five representative manipulation policies, we find that current policies can acquire simple instruction-grounding skills, but still struggle with fine-grained motor atoms, counting, and logical filtering. More importantly, strong atomic performance does not reliably transfer to held-out compositional tasks. ATOM-Bench provides a diagnostic testbed for studying whether failures arise from weak motor execution, poor instruction grounding, or limited compositional reuse.

Problem

• atomic skills
• compositional generalization
• manipulation policies
• real-world benchmark
• robotic control

Innovation

• atomic skills
• compositional generalization
• real-world benchmark
• manipulation policies
• foundation models