Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

2026-08-12

AI Summary

This work addresses the lack of systematic evaluation benchmarks for multimodal large language models (MLLMs) in scientific diagram understanding and generation. We propose Diagram-MMU, the first multimodal benchmark tailored to scientific research scenarios, encompassing 3.7k curated diagrams across six disciplines and 18.3k human-verified questions. It supports three core tasks: diagram-to-code generation, code editing, and diagram-based question answering, and introduces an agent interaction framework for holistic capability assessment. Experimental results reveal that diagram-to-code generation is significantly more challenging than question answering. While most models exhibit improved parsing capabilities in the agent setting, their question-answering performance declines; notably, only Claude-4.6 Opus consistently outperforms others across all tasks.

Abstract

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.

Problem

• scientific diagrams
• multimodal large language models
• diagram-to-code
• benchmark
• diagram understanding

Innovation

• scientific diagram understanding
• diagram-to-code generation
• multimodal benchmark
• multimodal LLM evaluation
• agentic reasoning