On Path to Multimodal Generalist: General-Level and General-Bench

2025-05-07

AI Summary

Existing evaluation benchmarks inadequately quantify the progress of multimodal large language models (MLLMs) toward human-level multimodal general intelligence. To address this, we propose the first systematic evaluation paradigm for multimodal general intelligence. Our method introduces: (1) General-Level, a novel five-tier taxonomy for quantifying multimodal generality; (2) Synergy, a core metric measuring cross-modal understanding–generation consistency and cooperative capability; and (3) General-Bench, a large-scale unified benchmark encompassing 700+ tasks and 325K samples, covering the broadest spectrum of modalities, formats, and capability dimensions to date. Applying this framework, we systematically evaluate over 100 state-of-the-art MLLMs, revealing a substantial gap between current models and true multimodal general intelligence. Our work establishes a quantitative, reproducible, and scalable assessment pathway toward artificial general intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Artificial General Intelligence

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

The Multimodal Large Language Model (MLLM) is currently experiencing rapid growth, driven by the advanced capabilities of LLMs. Unlike earlier specialists, existing MLLMs are evolving towards a Multimodal Generalist paradigm. Initially limited to understanding multiple modalities, these models have advanced to not only comprehend but also generate across modalities. Their capabilities have expanded from coarse-grained to fine-grained multimodal understanding and from supporting limited modalities to arbitrary ones. While many benchmarks exist to assess MLLMs, a critical question arises: Can we simply assume that higher performance across tasks indicates a stronger MLLM capability, bringing us closer to human-level AI? We argue that the answer is not as straightforward as it seems. This project introduces General-Level, an evaluation framework that defines 5-scale levels of MLLM performance and generality, offering a methodology to compare MLLMs and gauge the progress of existing systems towards more robust multimodal generalists and, ultimately, towards AGI. At the core of the framework is the concept of Synergy, which measures whether models maintain consistent capabilities across comprehension and generation, and across multiple modalities. To support this evaluation, we present General-Bench, which encompasses a broader spectrum of skills, modalities, formats, and capabilities, including over 700 tasks and 325,800 instances. The evaluation results that involve over 100 existing state-of-the-art MLLMs uncover the capability rankings of generalists, highlighting the challenges in reaching genuine AI. We expect this project to pave the way for future research on next-generation multimodal foundation models, providing a robust infrastructure to accelerate the realization of AGI. Project page: https://generalist.top/

Problem

• Evaluating MLLM performance and generality across multiple modalities
• Assessing synergy between comprehension and generation in MLLMs
• Developing a benchmark for robust multimodal generalist AI

Innovation

• Introduces General-Level evaluation framework for MLLMs
• Proposes Synergy concept for multimodal consistency
• Develops General-Bench with 700 tasks and 325,800 instances