MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents

2024-10-04

International Conference on Learning Representations

AI Summary

Existing MLLM-based agents retrieve multimodal trajectories relying on superficial text or visual similarity, neglecting their actual effectiveness for target tasks—thus limiting embodied agents’ cross-scenario generalization. This paper proposes treating the MLLM itself as a differentiable, interactive retriever, optimized via preference learning to select trajectories that are genuinely effective for unseen tasks. Our key contributions are: (1) the first end-to-end trainable retrieval paradigm using an MLLM as the retriever; and (2) an LLM-driven trajectory abstraction mechanism that preserves critical milestones while drastically reducing representational overhead. Evaluated across multiple embodied simulation environments, our method significantly improves task success rates on unseen scenes, outperforming conventional similarity-based retrieval baselines. Code and benchmark are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

MLLM agents demonstrate potential for complex embodied tasks by retrieving multimodal task-relevant trajectory data. However, current retrieval methods primarily focus on surface-level similarities of textual or visual cues in trajectories, neglecting their effectiveness for the specific task at hand. To address this issue, we propose a novel method, MLLM As ReTriever (MART), which enhances the performance of embodied agents by utilizing interaction data to fine-tune an MLLM retriever based on preference learning, such that the retriever fully considers the effectiveness of trajectories and prioritizes them for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that leverages MLLMs' summarization capabilities to represent trajectories with fewer tokens while preserving key information, enabling agents to better comprehend milestones in the trajectory. Experimental results across various environments demonstrate our method significantly improves task success rates in unseen scenes compared to baseline methods. This work presents a new paradigm for multimodal retrieval in embodied agents, by fine-tuning a general-purpose MLLM as the retriever to assess trajectory effectiveness. All the code for benchmark tasks, simulator modifications, and the MLLM retriever is available at https://github.com/PKU-RL/MART.

Problem

• Enhancing retrieval of task-relevant multimodal trajectory data
• Addressing surface-level similarity neglect in current retrieval methods
• Improving task success rates in unseen environments

Innovation

• MART fine-tunes MLLM retriever via preference learning
• Trajectory Abstraction reduces tokens while keeping key info
• MART improves task success in unseen scenes