Experience Transfer for Multimodal LLM Agents in Minecraft Game

2026-04-07

AI Summary

This work addresses the challenge that multimodal large language model agents struggle to efficiently reuse past experiences for solving novel tasks in complex game environments. To overcome this limitation, the authors propose Echo, a framework that explicitly models experience as transferable knowledge across five dimensions: structure, attributes, processes, functions, and interactions. Echo further introduces an In-Context Analogical Learning (ICAL) mechanism, enabling a shift from passive memory retrieval to active knowledge transfer. Evaluated on zero-shot object unlocking tasks in Minecraft, Echo demonstrates a 1.3–1.7× improvement in task efficiency and exhibits chain-unlocking capabilities, significantly enhancing the agent’s adaptability and learning effectiveness.

Abstract

Multimodal LLM agents operating in complex game environments must continually reuse past experience to solve new tasks efficiently. In this work, we propose Echo, a transfer-oriented memory framework that enables agents to derive actionable knowledge from prior interactions rather than treating memory as a passive repository of static records. To make transfer explicit, Echo decomposes reusable knowledge into five dimensions: structure, attribute, process, function, and interaction. This formulation allows the agent to identify recurring patterns shared across different tasks and infer what prior experience remains applicable in new situations. Building on this formulation, Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples. Experiments in Minecraft show that, under a from-scratch learning setting, Echo achieves a 1.3x to 1.7x speed-up on object-unlocking tasks. Moreover, Echo exhibits a burst-like chain-unlocking phenomenon, rapidly unlocking multiple similar items within a short time interval after acquiring transferable experience. These results suggest that experience transfer is a promising direction for improving the efficiency and adaptability of multimodal LLM agents in complex interactive environments.

Problem

• experience transfer
• multimodal LLM agents
• complex interactive environments
• task generalization
• memory reuse

Innovation

• experience transfer
• multimodal LLM agents
• memory framework
• In-Context Analogy Learning
• task generalization