InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback

2025-05-29

AI Summary

Current multimodal large language models (MLLMs) lack robust capabilities for sustained, interleaved multi-round multimodal interaction, hindering their ability to comprehend entangled cross-modal contexts and generate coherent, context-aware responses. Method: We propose InterMT—a first-of-its-kind preference dataset grounded in real human feedback, comprising 15.6k prompts, 52.6k dialogue instances, and 32.4k preference pairs—accompanied by a novel nine-dimensional global+local preference annotation protocol. We further introduce a tool-augmented agent workflow to automatically generate high-quality multi-round QA, establish an interleaved multimodal context modeling framework, and release the benchmark InterMT-Bench. Contribution/Results: Experiments reveal that judge model performance follows a power-law improvement with increasing dialogue rounds. Our open-sourced dataset and methodology significantly enhance MLLM alignment with human intent in multi-turn interactive settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As multimodal large models (MLLMs) continue to advance across challenging tasks, a key question emerges: What essential capabilities are still missing? A critical aspect of human learning is continuous interaction with the environment -- not limited to language, but also involving multimodal understanding and generation. To move closer to human-level intelligence, models must similarly support multi-turn, multimodal interaction. In particular, they should comprehend interleaved multimodal contexts and respond coherently in ongoing exchanges. In this work, we present an initial exploration through the InterMT -- the first preference dataset for multi-turn multimodal interaction, grounded in real human feedback. In this exploration, we particularly emphasize the importance of human oversight, introducing expert annotations to guide the process, motivated by the fact that current MLLMs lack such complex interactive capabilities. InterMT captures human preferences at both global and local levels into nine sub-dimensions, consists of 15.6k prompts, 52.6k multi-turn dialogue instances, and 32.4k human-labeled preference pairs. To compensate for the lack of capability for multi-modal understanding and generation, we introduce an agentic workflow that leverages tool-augmented MLLMs to construct multi-turn QA instances. To further this goal, we introduce InterMT-Bench to assess the ability of MLLMs in assisting judges with multi-turn, multimodal tasks. We demonstrate the utility of InterMT through applications such as judge moderation and further reveal the multi-turn scaling law of judge model. We hope the open-source of our data can help facilitate further research on aligning current MLLMs to the next step. Our project website can be found at https://pku-intermt.github.io .

Problem

• Lack of multi-turn multimodal interaction in MLLMs
• Need for human-aligned preference datasets in multimodal contexts
• Absence of tools to assess MLLMs' multi-turn multimodal capabilities

Innovation

• Multi-turn multimodal interaction dataset
• Tool-augmented MLLMs workflow
• Human oversight with expert annotations