Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

2026-06-22

AI Summary

This work addresses the gap between high-level semantic instructions and low-level motion control in zero-shot, language-guided robotic manipulation, particularly tackling the spatial ambiguity and semantic hallucination challenges of vision-language models in semi-structured environments. The authors propose a training-free, modular framework that decouples the visuomotor pipeline into three stages: visual perception, semantic interpretation, and task execution. It innovatively combines FastSAM with Set-of-Mark prompting to generate verifiable visual anchors and repurposes a unified foundation model as a pure language model for semantic routing—eliminating the need for fine-tuning or coordinate-based programming. Built upon a large language model and MoveIt Task Constructor, the end-to-end reconfigurable pipeline achieves a 62% zero-shot success rate on open-world sequential manipulation and dense relational reasoning tasks, demonstrating its effectiveness without domain-specific training.

Abstract

This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.

Problem

• zero-shot manipulation
• language-guided robotics
• semantic ambiguity
• spatial reasoning
• vision-language models

Innovation

• zero-shot manipulation
• modular framework
• vision-language grounding
• semantic routing
• task orchestration