🤖 AI Summary
Large language models (LLMs) exhibit significant performance degradation in mathematical reasoning when irrelevant information is present, and existing prompting methods can only detect—but not effectively suppress—such distractors. To address this, we introduce GSMIR, the first benchmark dataset for distractor identification and robust reasoning in elementary-school mathematics. We further propose ATF (Automatic Two-stage Framework), a novel prompting method that operates in two phases: first, semantic analysis via chain-of-thought reasoning and self-reflection; second, dynamic identification and filtering of irrelevant content. ATF is the first approach to jointly optimize distractor detection and adaptive mitigation, overcoming key limitations of conventional prompting techniques. Experiments on GSMIR demonstrate that ATF improves average accuracy by 23.6% across mainstream LLMs, substantially enhancing robustness against irrelevant information. This work establishes a new paradigm for trustworthy, interference-resilient reasoning in prompt engineering.
📝 Abstract
In recent years, Large language models (LLMs) have garnered significant attention due to their superior performance in complex reasoning tasks. However, recent studies may diminish their reasoning capabilities markedly when problem descriptions contain irrelevant information, even with the use of advanced prompting techniques. To further investigate this issue, a dataset of primary school mathematics problems containing irrelevant information, named GSMIR, was constructed. Testing prominent LLMs and prompting techniques on this dataset revealed that while LLMs can identify irrelevant information, they do not effectively mitigate the interference it causes once identified. A novel automatic construction method, ATF, which enhances the ability of LLMs to identify and self-mitigate the influence of irrelevant information, is proposed to address this shortcoming. This method operates in two steps: first, analysis of irrelevant information, followed by its filtering. The ATF method, as demonstrated by experimental results, significantly improves the reasoning performance of LLMs and prompting techniques, even in the presence of irrelevant information on the GSMIR dataset.