Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

2026-06-30

AI Summary

This work addresses the limitations of current vision-language model (VLM)-driven robotic grasping methods, which rely heavily on visual similarity while neglecting physical affordances—such as graspable regions and material fragility—and lack spatial reasoning and failure recovery capabilities, leading to poor generalization in dense, cluttered scenes. To overcome these challenges, the authors propose an agent framework that integrates retrieval-augmented generation (RAG) with VLMs, introducing four key innovations: a four-dimensional affordance descriptor, a hierarchical affordance-aware RAG module, a scene-graph-based spatial constraint reasoner, and a three-tier self-reflective retry mechanism covering 14 failure modes. This approach unifies functional affordances, spatial relationships, and closed-loop recovery within a VLM-based grasping system for the first time. Evaluated across 12 benchmarks spanning single grasps, interactive tasks, and long-horizon missions, the method achieves an overall success rate of 78.3%, representing an absolute improvement of 53.3 percentage points over pure VLM baselines.

Abstract

Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse. We present Agentic RAG-VLM, a unified framework that bridges VLM-based semantic understanding and physically grounded grasp execution by integrating retrieval-augmented generation (RAG) with vision-language models (VLMs) and agentic self-reflective planning. Agentic RAG-VLM introduces three tightly coupled components: (1) a Hierarchical Affordance-Aware RAG (HAA-RAG) that encodes four-dimensional affordance descriptors, including type, material, fragility, and graspable region, and retrieves strategies by functional affordance compatibility rather than visual appearance; (2) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and (3) an Agentic Self-Reflective Pipeline with a 14-type failure taxonomy and three-level adaptive retry for closed-loop grasp refinement. Evaluated on a 12-task benchmark spanning single-grasp, interactive, and long-horizon scenarios with 360 trials per configuration, Agentic RAG-VLM achieves 78.3 percent overall success, a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating that affordance-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation.

Problem

• robotic grasping
• visual-language models
• affordance awareness
• cluttered environments
• failure recovery

Innovation

• Affordance-Aware Retrieval
• Scene Graph Reasoning
• Self-Reflective Planning
• Robotic Grasping
• Retrieval-Augmented Generation