PhysClaw-0: A Symbiotic Agentic System for Robot Autonomy via Language Corrections

2026-07-15

AI Summary

This work addresses the escalating human supervision cost in robotic autonomous data collection, where repeated human corrections for recurring failure modes lead to linearly increasing labor demands over task duration. To overcome this limitation, the authors introduce PhysClaw-0, a human–robot symbiotic agent that enables cross-episode memory and reuse of natural language corrections for the first time. By leveraging a large language model to parse human instructions into structured adjustment policies, integrating a vision-language model as a verification module, and incorporating an autonomous reset mechanism, PhysClaw-0 forms a closed-loop data collection system that solicits human intervention only after exhausting retry attempts. Evaluated on a real-world tabletop clearing task, the approach reduces human effort to 16% of baseline levels, boosts single-attempt success rates from 12.5% to 47.5%, and achieves fine-tuning performance comparable to teleoperation-based training, substantially lowering long-term human labor costs.

Abstract

Autonomous data collection governs the volume and quality of real-world trajectories for manipulation policy learning. Existing pipelines reduce human effort via self-resetting, VLM verification, or language-guided correction, yet episode-scoped fixes must be reissued whenever the same failure recurs, so oversight cost grows with session length rather than with the number of distinct problems. We present PhysClaw-0, a human-robot symbiotic agentic system in which corrections are retained and reused across rounds. The collection loop collects, verifies, and resets autonomously, pausing for a remote operator only when a phase exhausts an explicit retry budget. An LLM parser maps each natural-language utterance to a structured adjustment stored in Corrective Memory, so addressed failure modes typically need not be corrected again under the same conditions. On a real-robot desktop-clearing testbed, PhysClaw-0 matches teleoperation episode success while reducing human working time to 16%. Language corrections improve verifier-human agreement in all four evaluated settings and raise average single-attempt success from 12.5% to 47.5% (arm-selection: 20.0% to 50.0%). Policies fine-tuned on PhysClaw-0 data match teleoperation-trained policy success at a fraction of collection human cost.

Problem

• robot autonomy
• language corrections
• human oversight cost
• failure recurrence
• autonomous data collection

Innovation

• Corrective Memory
• Language-guided Correction
• Symbiotic Agentic System
• Autonomous Data Collection
• LLM-based Parsing