From Code to Action: Hierarchical Learning of Diffusion-VLM Policies

2025-09-29

AI Summary

To address poor generalization in imitation learning and scarcity of long-horizon task data for robotic manipulation, this paper proposes a hierarchical imitation learning framework. At the high level, a code-generation visual-language model (VLM), grounded in open-source robot APIs, performs semantic-driven task decomposition and planning; generated subtask functions serve as interpretable, structured supervision signals. At the low level, a diffusion-based policy models continuous actions and incorporates a memory mechanism to capture non-Markovian dependencies. Crucially, the framework enables decoupled training and independent evaluation of high-level planning and low-level control. Experiments demonstrate that, compared to flat policies, our approach achieves significantly higher success rates on complex long-horizon tasks—such as object swapping—while exhibiting improved robustness and cross-task generalization.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Imitation learning for robotic manipulation often suffers from limited generalization and data scarcity, especially in complex, long-horizon tasks. In this work, we introduce a hierarchical framework that leverages code-generating vision-language models (VLMs) in combination with low-level diffusion policies to effectively imitate and generalize robotic behavior. Our key insight is to treat open-source robotic APIs not only as execution interfaces but also as sources of structured supervision: the associated subtask functions - when exposed - can serve as modular, semantically meaningful labels. We train a VLM to decompose task descriptions into executable subroutines, which are then grounded through a diffusion policy trained to imitate the corresponding robot behavior. To handle the non-Markovian nature of both code execution and certain real-world tasks, such as object swapping, our architecture incorporates a memory mechanism that maintains subtask context across time. We find that this design enables interpretable policy decomposition, improves generalization when compared to flat policies and enables separate evaluation of high-level planning and low-level control.

Problem

• Addresses imitation learning generalization in robotic manipulation
• Leverages code-generating VLMs with diffusion policies hierarchically
• Incorporates memory for non-Markovian tasks like object swapping

Innovation

• Hierarchical framework combining code-generating VLMs with diffusion policies
• Using robotic APIs as structured supervision for semantic labeling
• Memory mechanism handling non-Markovian tasks through subtask context