SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

2026-08-26

AI Summary

本文通过构建SciMIF基准，采用系统性指令注入方法增强科学数据集，评估多模态大语言模型在遵循复杂科学指令方面的能力，揭示了不同学科间性能差异。

Abstract

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .

Problem

• Multimodal Large Language Models
• Scientific Domains
• Instruction Following
• Constraint Adherence

Innovation

• SciMIF
• Multimodal Large Language Models (MLLMs)
• scientific instruction following
• constraint groups
• instruction injection pipeline