LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

2026-05-14

AI Summary

This work addresses the instability of large language models on logically equivalent problem variants that differ only in entity substitutions—a vulnerability inadequately captured by existing evaluation protocols due to their lack of systematicity. To overcome the limitations of random sampling, the authors propose the Logic-Preserving Difficulty Scaling (LPDS) framework, which introduces a systematic search strategy coupled with quantitative difficulty metrics to actively identify high-difficulty variants most likely to induce model failures. Experiments demonstrate that variants uncovered by LPDS degrade model performance up to five times more severely than those found via random sampling. Furthermore, fine-tuning on these challenging variants substantially and consistently enhances model robustness. The LPDS framework thus markedly improves the sensitivity and effectiveness of robustness evaluation for large language models.

Abstract

As large language models (LLMs) are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustly. In particular, a model that can solve a given problem should not fail simply because certain entities$\unicode{x2013}$such as names, numbers, or other contextual details$\unicode{x2013}$have changed while the underlying problem logic remains the same. Prior work suggests that current LLMs still struggle with this form of robustness: they often succeed on some variations of a problem but fail on others. However, existing evaluations often lack a systematic way to identify which logic-preserving variations are most likely to induce failure. Instead, they typically test a random subset of allowable variations, which can overstate robustness. To address this gap, we introduce logic-preserving difficulty scaling (LPDS), a framework that (i) quantifies the difficulty of a problem variation and (ii) systematically searches the space of allowable variations to find those that maximize difficulty and expose failures. We show that as difficulty increases, performance declines and errors in the models' reasoning chains become more pronounced. We further demonstrate that LPDS efficiently finds difficult problem variations for a model, resulting in performance drops up to 5 times larger compared to random sampling. Finally, we show that fine-tuning on more difficult variations leads to more consistent robustness gains than training on easier ones.

Problem

• LLM robustness
• logic-preserving variations
• difficulty scaling
• systematic evaluation
• reasoning consistency

Innovation

• logic-preserving
• difficulty scaling
• LLM robustness
• systematic evaluation
• adversarial variation