Key-Gram: Extensible World Knowledge for Embodied Manipulation

2026-05-18

AI Summary

This work addresses the limitations of existing embodied control models, which often tightly couple language and vision, leading to modality competition and poor scalability in world knowledge. To overcome this, the authors propose Key-Gram, a novel framework that explicitly decouples linguistic knowledge from visual reasoning for the first time. Key-Gram stores task instructions in an external, extensible memory module and employs deterministic hash-based retrieval to achieve O(1) lookup complexity. Context-aware gating and a lightweight convolutional fusion mechanism then inject retrieved knowledge into the backbone network. This architecture enables partitioned training and zero-shot transfer without target-domain fine-tuning. Evaluated on RoboTwin2.0, LIBERO-Plus, and real-world dual-arm tasks, Key-Gram achieves an average performance improvement of 35.8% over strong baselines, demonstrating superior generalization and efficiency.

Abstract

Embodied control increasingly requires models to follow compositional language instructions while reasoning over dynamic visual states. However, current vision-language-action policies and world-action models often couple linguistic knowledge with visual computation in a shared backbone or conditioning pathway, leading to modality competition and making knowledge extension dependent on backbone updates. In this paper, we introduce Key-Gram, a conditional-memory framework that separates language-derived world knowledge from visual-state reasoning for embodied control. At its core is a memory module that decomposes an instruction into task-specific key-grams, retrieves static linguistic priors through deterministic hashed lookup, and injects the retrieved entries into selected hidden layers through context-aware gating and lightweight convolutional fusion. This design allows the backbone to devote its main capacity to visual reasoning and action inference, while reusable instruction knowledge is stored in an extensible external memory. The logical memory table can be conveniently partitioned during training and, due to its $O(1)$ lookup pattern, efficiently placed on host memory during inference. Across RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, Key-Gram consistently improves both $π_{0}$ and $π_{0.5}$ backbones, with average relative gains of $29.5\%/9.9\%$ on RoboTwin2.0, $35.8\%/4.5\%$ on LIBERO-Plus transfer without target-domain fine-tuning, and $15.4\%/8.1\%$ on real-world long-horizon tasks. These results demonstrate that externalized linguistic memory provides an effective and extensible mechanism for improving compositional grounding, transfer, and real-world manipulation.

Problem

• embodied manipulation
• vision-language-action policies
• modality competition
• knowledge extension
• compositional grounding

Innovation

• Key-Gram
• external memory
• compositional grounding
• vision-language-action
• embodied manipulation