Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams
Large language models exhibit significant limitations in multimodal reasoning on Chemistry Olympiad problems—comprising molecular structure diagrams, chemical notation, and textual inference—due to poor cross-modal alignment and visual grounding. Method: We introduce the first dedicated multimodal benchmark for Chemistry Olympiads, built on authentic USNCO exam questions with fine-grained structural annotations, and conduct systematic evaluation of 40 open- and closed-source multimodal large models. We identify a pervasive “multimodal fusion failure” phenomenon—where removing images paradoxically improves accuracy—and propose a Chain-of-Thought (CoT) prompting strategy to enhance visual localization and reasoning consistency, validated via occlusion-based interpretability analysis. Contribution/Results: Experiments reveal that state-of-the-art models (e.g., GPT-5, Gemini 2.5 Pro) achieve sub-50% average accuracy; CoT prompting boosts accuracy by 12.3%. This work establishes a new scientific multimodal reasoning benchmark, uncovers a novel failure mode, and provides a transferable optimization framework for domain-specific multimodal understanding.