CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps

2026-07-12

AI Summary

Current large language models often rely on internal memorization or dataset-specific shortcuts in multi-hop question answering, lacking genuine context-based reasoning capabilities. To address this limitation, this work proposes the CRiT-QA dataset, which innovatively integrates counterfactual entity substitution with multi-anchor distractor chains to construct a context-constrained evaluation framework for multi-hop reasoning. This design compels models to perform complete inference strictly based on provided evidence, thereby effectively suppressing shallow heuristic strategies. Empirical evaluations on mainstream large language models reveal significant vulnerabilities in their reasoning under counterfactual conditions and distractor traps, highlighting the dataset’s utility in enabling more reliable assessment of true multi-hop reasoning abilities.

Abstract

Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge. Although current models achieve strong results on existing multi-hop question answering datasets, such performance often masks two critical vulnerabilities: (1) reliance on internal parametric knowledge rather than adherence to the provided context, and (2) exploitation of dataset shortcuts, such as single-document cues or type-matching, that diminish the need for genuine evidence aggregation across multiple documents. We introduce CRiT-QA (Counterfactual Reasoning with Traps), a dataset explicitly designed to address both limitations. To neutralize reliance on memorized knowledge and enforce strict context dependency, CRiT-QA transforms factual reasoning chains with counterfactual entities. Furthermore, it injects multi-anchor distractor chains, plausible but incorrect reasoning paths that diverge at different hops. These traps require models to follow the entire reasoning process rather than exploiting shallow heuristics. Our experiments show that LLMs exhibit substantial performance degradation on CRiT-QA compared to standard datasets, exposing their vulnerability to counterfactual conditions and distractor traps. CRiT-QA thus serves as a rigorous diagnostic tool for evaluating genuine multi-hop reasoning and provides a foundation for developing more reliable, evidence-grounded LLMs.

Problem

• multi-hop reasoning
• large language models
• counterfactual reasoning
• distractor traps
• context dependency

Innovation

• counterfactual reasoning
• multi-hop QA
• distractor traps
• context dependency
• reasoning robustness