From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?

2026-03-03

AI Summary

This work investigates the feasibility of employing large language models (LLMs) as core cognitive controllers for embodied agents, aiming to bridge the gap between high-level linguistic instructions and low-level perception-action loops. To this end, the authors propose an LLM-driven cognitive architecture that integrates working memory and episodic memory, enabling a mobile manipulator to execute tasks in simulated household environments through high-level tool interfaces such as navigation, grasping, and placing. Experimental results demonstrate that the approach exhibits emergent reasoning and memory-guided adaptive planning capabilities in structured object manipulation tasks. This study provides the first systematic validation of LLMs’ potential in embodied intelligence, while also uncovering critical challenges including task hallucination and insufficient instruction following.

Technology Category

• Intelligent Robots: Embodied AI
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In order to flexibly act in an everyday environment, a robotic agent needs a variety of cognitive capabilities that enable it to reason about plans and perform execution recovery. Large language models (LLMs) have been shown to demonstrate emergent cognitive aspects, such as reasoning and language understanding; however, the ability to control embodied robotic agents requires reliably bridging high-level language to low-level functionalities for perception and control. In this paper, we investigate the extent to which an LLM can serve as a core component for planning and execution reasoning in a cognitive robot architecture. For this purpose, we propose a cognitive architecture in which an agentic LLM serves as the core component for planning and reasoning, while components for working and episodic memories support learning from experience and adaptation. An instance of the architecture is then used to control a mobile manipulator in a simulated household environment, where environment interaction is done through a set of high-level tools for perception, reasoning, navigation, grasping, and placement, all of which are made available to the LLM-based agent. We evaluate our proposed system on two household tasks (object placement and object swapping), which evaluate the agent's reasoning, planning, and memory utilisation. The results demonstrate that the LLM-driven agent can complete structured tasks and exhibits emergent adaptation and memory-guided planning, but also reveal significant limitations, such as hallucinations about the task success and poor instruction following by refusing to acknowledge and complete sequential tasks. These findings highlight both the potential and challenges of employing LLMs as embodied cognitive controllers for autonomous robots.

Problem

• embodied cognition
• large language models
• robotic agents
• cognitive architecture
• language-to-action

Innovation

• LLM-based agent
• embodied cognition
• cognitive architecture
• memory-guided planning
• robotic reasoning