Grounding Language Models with Semantic Digital Twins for Robotic Planning

2025-06-19

AI Summary

This work addresses the challenges of adaptability and robustness in robotic execution of natural language instructions within dynamic environments. We propose a deep synergy framework integrating large language models (LLMs) with semantic digital twins (SDTs). Methodologically, we introduce the first bidirectional coupling: the SDT provides real-time, semantically grounded environmental representations and affordance-aware perception, while the LLM performs instruction parsing, reflective reasoning, and generation of structured action triplets—augmented with failure-driven iterative replanning. Our technical contributions include SDT modeling, adaptation to the ALFRED benchmark, and closed-loop simulation evaluation. Experiments demonstrate significant improvements in task success rate and fault tolerance across diverse household scenarios, effectively handling uncertainties such as object pose variations, occlusions, and execution failures. The framework establishes a novel paradigm for semantic-level task planning in embodied intelligence.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Embodied AI
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We introduce a novel framework that integrates Semantic Digital Twins (SDTs) with Large Language Models (LLMs) to enable adaptive and goal-driven robotic task execution in dynamic environments. The system decomposes natural language instructions into structured action triplets, which are grounded in contextual environmental data provided by the SDT. This semantic grounding allows the robot to interpret object affordances and interaction rules, enabling action planning and real-time adaptability. In case of execution failures, the LLM utilizes error feedback and SDT insights to generate recovery strategies and iteratively revise the action plan. We evaluate our approach using tasks from the ALFRED benchmark, demonstrating robust performance across various household scenarios. The proposed framework effectively combines high-level reasoning with semantic environment understanding, achieving reliable task completion in the face of uncertainty and failure.

Problem

• Integrate SDTs and LLMs for robotic task execution
• Decompose language instructions into structured action triplets
• Generate recovery strategies for execution failures

Innovation

• Integrates Semantic Digital Twins with LLMs
• Decomposes instructions into structured action triplets
• Uses error feedback for iterative plan revision