Fail-RAG : A Retrieval Augmented Generation Informed Framework for Robot Failure Identification

2026-06-17

AI Summary

This work addresses the challenge of effectively identifying unexpected failures in warehouse robots operating in dynamic environments, where task and scene variations often render traditional rule-based methods inadequate. To this end, the paper proposes Fail-RAG, a novel retrieval-augmented generation (RAG)-based framework for robot fault detection. Fail-RAG introduces RAG into robotic failure diagnosis for the first time, integrating multimodal embeddings from visual and contextual data, similarity-based retrieval, and instruction-tuned visual language models (VLMs) to enable context-aware, fine-grained fault understanding. Experimental results across five representative manipulation tasks in both simulated and real-world settings demonstrate that Fail-RAG improves average fault detection accuracy by 25 percentage points over off-the-shelf VLMs, substantially enhancing adaptability to dynamic failure scenarios.

Abstract

Industry automation is witnessing an evolution in robotics driven by both technological breakthroughs and societal changes: progress towards generalist robots, embodied and physical artificial intelligence (AI), and increasing labor shortage in manufacturing.An intelligent autonomous robot needs to not only act according to planned motions but also react to any unexpected events. In this study, we focus on such unexpected events in warehouses where robots are used for material handling. Specifically, we refer to any unexpected events as failures and develop methods to detect robot operations related failures. Rule-based detection methods may break since the form of failures could change due to the dynamic nature of both environments and tasks. We propose 'Fail-RAG', a Retrieval Augmented Generation (RAG)-based failure detection framework where failure images and context information are embedded and queried against a failure database by calculating their similarities. Vision-Language Models (VLMs) are further used to analyze failures and provide details by following our instruction template. We evaluated the performance of Fail-RAG by conducting both simulation and physical experiments using fixed robot arms and a mobile manipulator for multiple tasks that are common in warehouse automation. Fail-RAG achieved 25 percentage point higher failure detection accuracy on average across five types of robot operations compared to using off-the-shelf VLMs, indicating its effectiveness for real-world failure detection.

Problem

• robot failure identification
• warehouse automation
• unexpected events
• failure detection
• material handling

Innovation

• Retrieval Augmented Generation
• Failure Detection
• Vision-Language Models
• Robot Autonomy
• Warehouse Automation