Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

2025-08-14

AI Summary

Multimodal large language models (MLLMs) face critical limitations in low-quality multimodal data, poor generalization on complex tasks, and inadequate evaluation frameworks—hindering their reliable deployment. This paper presents the first systematic survey of tool-augmented MLLMs, organized along four dimensions: data construction, task enhancement, evaluation methodology, and future challenges. We propose a unified collaborative framework integrating multimodal encoders, large language models, and external tools—including APIs, domain-specific expert models, and knowledge bases—to significantly improve cross-modal understanding and reasoning. Furthermore, we develop an open-source toolkit repository to empirically validate diverse tool-augmentation strategies. Our work provides both theoretical foundations and practical guidelines for high-fidelity multimodal data generation, robust complex-scenario problem solving, and trustworthy evaluation of MLLMs—advancing multimodal AI toward greater reliability, interpretability, and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

By integrating the perception capabilities of multimodal encoders with the generative power of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs), exemplified by GPT-4V, have achieved great success in various multimodal tasks, pointing toward a promising pathway to artificial general intelligence. Despite this progress, the limited quality of multimodal data, poor performance on many complex downstream tasks, and inadequate evaluation protocols continue to hinder the reliability and broader applicability of MLLMs across diverse domains. Inspired by the human ability to leverage external tools for enhanced reasoning and problem-solving, augmenting MLLMs with external tools (e.g., APIs, expert models, and knowledge bases) offers a promising strategy to overcome these challenges. In this paper, we present a comprehensive survey on leveraging external tools to enhance MLLM performance. Our discussion is structured along four key dimensions about external tools: (1) how they can facilitate the acquisition and annotation of high-quality multimodal data; (2) how they can assist in improving MLLM performance on challenging downstream tasks; (3) how they enable comprehensive and accurate evaluation of MLLMs; (4) the current limitations and future directions of tool-augmented MLLMs. Through this survey, we aim to underscore the transformative potential of external tools in advancing MLLM capabilities, offering a forward-looking perspective on their development and applications. The project page of this paper is publicly available athttps://github.com/Lackel/Awesome-Tools-for-MLLMs.

Problem

• Enhancing MLLMs with external tools for better performance
• Improving multimodal data quality and annotation processes
• Addressing evaluation challenges in MLLM reliability and applicability

Innovation

• Integrate multimodal encoders with LLMs
• Augment MLLMs using external tools
• Survey tool-enhanced MLLM performance