UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark

2026-03-05

AI Summary

This work addresses the lack of unified understanding and generation capabilities in existing models for arbitrarily composed, interleaved multimodal inputs and outputs. To this end, we introduce UniM, the first arbitrary-to-arbitrary interleaved multimodal benchmark, encompassing seven modalities—text, image, audio, video, document, code, and 3D—spanning 30 domains and comprising 31K high-quality samples, along with a multidimensional evaluation framework. Building upon a multimodal large language model architecture, we propose the baseline model UniMA, which incorporates a traceable reasoning mechanism to jointly model semantic understanding, structural integrity, and interleaved coherence. Experimental results demonstrate that UniM presents a significant challenge, revealing critical bottlenecks in current models’ ability to handle unified interleaved multimodal tasks and thereby establishing a foundation and direction for future research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In real-world multimodal applications, systems usually need to comprehend arbitrarily combined and interleaved multimodal inputs from users, while also generating outputs in any interleaved multimedia form. This capability defines the goal of any-to-any interleaved multimodal learning under a unified paradigm of understanding and generation, posing new challenges and opportunities for advancing Multimodal Large Language Models (MLLMs). To foster and benchmark this capability, this paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset. UniM contains 31K high-quality instances across 30 domains and 7 representative modalities: text, image, audio, video, document, code, and 3D, each requiring multiple intertwined reasoning and generation capabilities. We further introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness&Generation Quality, Response Structure Integrity, and Interleaved Coherence. In addition, we propose UniMA, an agentic baseline model equipped with traceable reasoning for structured interleaved generation. Comprehensive experiments demonstrate the difficulty of UniM and highlight key challenges and directions for advancing unified any-to-any multimodal intelligence. The project page is https://any2any-mllm.github.io/unim.

Problem

• any-to-any multimodal learning
• interleaved multimodal inputs
• multimodal benchmark
• unified multimodal understanding and generation
• Multimodal Large Language Models

Innovation

• any-to-any multimodal learning
• interleaved multimodal benchmark
• multimodal large language models
• structured interleaved generation
• multimodal evaluation suite