ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation

2026-08-21

AI Summary

为解决现有化学基准测试对模型能力评估不全面的问题，本文提出ChemDIRT框架，通过多样化指令、表示和任务来更可靠地评估大语言模型在化学推理上的鲁棒性。

Abstract

The rapid advancement of large language models (LLMs) has led to increasing interest in their application to scientific domains such as chemistry. However, existing chemistry benchmarks often provide only a narrow view of model capability, focusing on limited task sets while overlooking robustness to variations in problem formulation and chemical representation. As a result, reported performance may overestimate a model's true ability to reason consistently across realistic settings. To address this challenge, we introduce ChemDIRT (Diversified Instruction, Representation, and Task Benchmark), a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in LLMs. ChemDIRT systematically measures model performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks. By evaluating both accuracy and consistency under these controlled perturbations, ChemDIRT provides a more reliable assessment of model reasoning capabilities than conventional single-format benchmarks. We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.

Problem

• chemistry benchmarks
• model capability
• robustness
• chemical reasoning
• LLMs

Innovation

• ChemDIRT
• Diversified Instruction
• Molecular Representations
• Robustness Evaluation
• Chemistry-LLM