A Survey on Agentic Multimodal Large Language Models

2025-10-13

AI Summary

Current AI agent research is shifting from static, passive, and domain-specific paradigms toward dynamic, proactive, and general-purpose architectures. Method: This paper introduces Agentic Multimodal Large Language Models (Agentic MLLMs) as a novel paradigm, formally defining its conceptual foundations and core characteristics. We propose the first three-dimensional theoretical framework—comprising internal intelligence (integrating reasoning, reflection, and memory), tool utilization, and environment interaction—that rigorously distinguishes Agentic MLLMs from conventional multimodal LLM agents. To enable empirical advancement, we open-source a training framework, a multimodal task dataset, an end-to-end evaluation benchmark, and a knowledge base. Results: Experiments demonstrate that our framework supports long-horizon planning and goal-directed behavior, significantly improving cross-task generalization. It establishes a scalable, foundational agent architecture toward Artificial General Intelligence.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With the recent emergence of revolutionary autonomous agentic systems, research community is witnessing a significant shift from traditional static, passive, and domain-specific AI agents toward more dynamic, proactive, and generalizable agentic AI. Motivated by the growing interest in agentic AI and its potential trajectory toward AGI, we present a comprehensive survey on Agentic Multimodal Large Language Models (Agentic MLLMs). In this survey, we explore the emerging paradigm of agentic MLLMs, delineating their conceptual foundations and distinguishing characteristics from conventional MLLM-based agents. We establish a conceptual framework that organizes agentic MLLMs along three fundamental dimensions: (i) Agentic internal intelligence functions as the system's commander, enabling accurate long-horizon planning through reasoning, reflection, and memory; (ii) Agentic external tool invocation, whereby models proactively use various external tools to extend their problem-solving capabilities beyond their intrinsic knowledge; and (iii) Agentic environment interaction further situates models within virtual or physical environments, allowing them to take actions, adapt strategies, and sustain goal-directed behavior in dynamic real-world scenarios. To further accelerate research in this area for the community, we compile open-source training frameworks, training and evaluation datasets for developing agentic MLLMs. Finally, we review the downstream applications of agentic MLLMs and outline future research directions for this rapidly evolving field. To continuously track developments in this rapidly evolving field, we will also actively update a public repository at https://github.com/HJYao00/Awesome-Agentic-MLLMs.

Problem

• Surveying dynamic AI agents using multimodal large language models
• Establishing conceptual framework for agentic intelligence and tool usage
• Compiling training resources for developing proactive generalizable AI systems

Innovation

• Agentic MLLMs enable long-horizon planning through reasoning
• Models proactively invoke external tools for problem-solving
• Agentic systems interact with environments to adapt strategies