Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

2026-03-17

AI Summary

This work addresses the limited ability of current large language models to support fine-grained evaluation of intermediate reasoning steps in multi-hop reasoning tasks, which hinders precise failure diagnosis. To this end, the authors introduce Omanic, an open-domain multi-hop question answering resource that includes two novel datasets—OmanicBench and OmanicSynth—featuring large-scale human-annotated and synthetically generated step-level labels. These datasets enable both explainable evaluation and supervised training of reasoning processes. Experimental results show that leading models achieve only 73.11% accuracy on OmanicBench. Furthermore, supervised fine-tuning using OmanicSynth combined with chain-of-thought (CoT) prompting yields an average improvement of 7.41 points across six reasoning and mathematical benchmarks, substantially enhancing multi-hop reasoning performance.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reasoning-focused large language models (LLMs) have advanced in many NLP tasks, yet their evaluation remains challenging: final answers alone do not expose the intermediate reasoning steps, making it difficult to determine whether a model truly reasons correctly and where failures occur, while existing multi-hop QA benchmarks lack step-level annotations for diagnosing reasoning failures. To address this gap, we propose Omanic, an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes. It contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench). Systematic evaluations show that state-of-the-art LLMs achieve only 73.11% multiple-choice accuracy on OmanicBench, confirming its high difficulty. Stepwise analysis reveals that CoT's performance hinges on factual completeness, with its gains diminishing under knowledge gaps and errors amplifying in later hops. Additionally, supervised fine-tuning on OmanicSynth brings substantial transfer gains (7.41 average points) across six reasoning and math benchmarks, validating the dataset's quality and further supporting the effectiveness of OmanicSynth as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.

Problem

• multi-hop reasoning
• large language models
• reasoning evaluation
• step-wise analysis
• intermediate reasoning steps

Innovation

• step-wise evaluation
• multi-hop reasoning
• intermediate reasoning annotation
• reasoning diagnosis
• supervised fine-tuning