Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference

2025-09-01

AI Summary

To address the fragility of skill-effect models under out-of-distribution conditions in long-horizon manipulation tasks, this paper proposes a failure-driven robustification framework. Our method leverages Stein variational inference (SVI) to synthesize semantically consistent and diverse simulated failure trajectories from real-robot failure data, thereby constructing a high-fidelity, targeted failure dataset. This dataset is integrated with skill-effect modeling and sim-to-real transfer to enable iterative model refinement. We evaluate the approach on complex mobile manipulation tasks—including multi-object transport and shelf organization—demonstrating substantial reductions in future failure rates compared to multiple baselines. Moreover, the framework exhibits strong generalization across different tasks and varying numbers of objects, without requiring task-specific retraining. Key contributions include: (i) the first application of SVI to failure-data augmentation for robotic skill learning; (ii) a principled, simulation-informed strategy for robustifying skill-effect models; and (iii) empirical validation of cross-task and cross-scale generalizability in real-world manipulation settings.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Skill effect models for long-horizon manipulation tasks are prone to failures in conditions not covered by training data distributions. Therefore, enabling robots to reason about and learn from failures is necessary. We investigate the problem of efficiently generating a dataset targeted to observed failures. After fine-tuning a skill effect model on this dataset, we evaluate the extent to which the model can recover from failures and minimize future failures. We propose Fail2Progress, an approach that leverages Stein variational inference to generate multiple simulation environments in parallel, enabling efficient data sample generation similar to observed failures. Our method is capable of handling several challenging mobile manipulation tasks, including transporting multiple objects, organizing a constrained shelf, and tabletop organization. Through large-scale simulation and real-world experiments, we demonstrate that our approach excels at learning from failures across different numbers of objects. Furthermore, we show that Fail2Progress outperforms several baselines.

Problem

• Learning from robot failures to improve manipulation skills
• Generating targeted datasets for failure recovery in robots
• Enhancing skill effect models to minimize future failures

Innovation

• Stein variational inference for failure simulation
• Parallel environment generation for targeted data
• Fine-tuning skill models with failure datasets