RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

2026-04-16

AI Summary

This work addresses the challenge that existing methods struggle to interpret multimodal user instructions and generalize to unseen tools. To overcome this limitation, the authors propose a retrieval-based framework for open-world multimodal tool selection. The approach first leverages a multimodal large language model to translate user queries into structured task descriptions, then retrieves the most suitable tool from a standardized repository via semantic matching. Notably, the framework supports seamless integration of new tools without retraining and incorporates Direct Preference Optimization (DPO) to enhance alignment between tasks and tools. Experimental results demonstrate that the proposed method significantly improves tool selection accuracy in open-world multimodal settings. The study also introduces the first benchmark dataset dedicated to this problem.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face model cards. Extensive experiments demonstrate that our approach significantly improves tool-selection performance, particularly in open-world, multimodal scenarios.

Problem

• multimodal tool use
• open-world generalization
• tool selection
• foundation models
• retrieval-based

Innovation

• retrieval-based tool selection
• multimodal large language models
• open-world generalization
• structured task description
• Direct Preference Optimization