ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

2026-04-17

AI Summary

This work addresses the significant limitations of multimodal large language models in reasoning about complex topological structures in diagrams—such as branching, merging, and cyclic dependencies—which existing benchmarks fail to adequately evaluate. To bridge this gap, we introduce ReactBench, the first benchmark specifically designed to assess topological reasoning in scientific diagrams, comprising 1,618 expert-annotated question-answer pairs derived from chemical reaction graphs. ReactBench evaluates models across four hierarchical tasks that probe both local recognition and global structural reasoning capabilities. Through controlled ablation studies disentangling perception from reasoning, we systematically evaluate 17 state-of-the-art models and find their performance on holistic structural tasks lags over 30% behind anchor-based tasks, revealing that reasoning—not perception—is the primary bottleneck. This benchmark establishes a new direction and evaluation framework for advancing visual structural understanding.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies, their reasoning capabilities degrade sharply, even on tasks as basic as counting endpoints. Existing benchmarks fail to probe this gap, focusing on semantic comprehension rather than structural reasoning. We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams. These real-world scientific diagrams offer an ideal testbed because they naturally span diverse structures from linear chains to cyclic graphs, while requiring both precise local recognition and coherent global reasoning. Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions. Extensive evaluation across 17 MLLMs reveals a significant performance gap exceeding 30% between anchor-based tasks and holistic structural reasoning tasks. Controlled ablations confirm this bottleneck lies in reasoning, not perception. These findings expose a fundamental deficit in structural understanding and establish directions for advancing visual reasoning.

Problem

• topological reasoning
• structural reasoning
• chemical reaction diagrams
• multimodal large language models
• benchmark

Innovation

• topological reasoning
• chemical reaction diagrams
• multimodal large language models
• structural understanding
• benchmark