VC Search: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning

2024-06-07

AI Summary

Existing LLM-based mathematical reasoning evaluations predominantly rely on idealized, well-defined benchmarks, overlooking ill-defined problems—common in real-world scenarios—that involve missing or contradictory constraints. Method: We introduce PMC, the first large-scale benchmark of ill-posed mathematical problems (5,000+ instances), and propose VCSEARCH, a training-free variable-constraint search framework. VCSEARCH automatically models problem structure via formal language parsing to enable zero-shot ill-posedness detection, breaks the accuracy–rejection trade-off, and enhances formal representation and solvability assessment of complex contradictory conditions. Contribution/Results: On diverse LLMs, VCSEARCH improves identification accuracy of unsolvable problems by ≥12% over baselines, significantly enhancing robustness and trustworthiness of mathematical reasoning under realistic, imperfect conditions.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have demonstrated impressive performance on reasoning tasks, including mathematical reasoning. However, the current evaluation mostly focuses on carefully constructed benchmarks and neglects the consideration of real-world reasoning problems that present missing or contradictory conditions, known as ill-defined problems. To further study this problem, we develop a largescale benchmark called Problems with Missing and Contradictory conditions ( PMC) containing over 5,000 validated ill-defined mathematical problems. Our preliminary experiments through PMC reveal two challenges about existing methods: (1) traditional methods exhibit a trade-off between solving accuracy and rejection capabilities, and (2) formal methods struggle with modeling complex problems. To address these challenges, We develop Variable-Constraint Search (VCSEARCH), a trainingfree framework that leverages formal language to detect ill-defined problems, where a variableconstraint pair search strategy is incorporated to improve the modeling capability of formal language. Extensive experiments demonstrate that VCSEARCH improves the accuracy of identifying unsolvable problems by at least 12% across different LLMs, thus achieving stronger robust mathematical reasoning ability.

Problem

• Addresses ill-defined mathematical reasoning problems
• Develops a framework to detect unsolvable problems
• Improves accuracy in identifying unsolvable conditions

Innovation

• VCSEARCH framework enhances mathematical reasoning
• Variable-constraint pair detects ill-defined problems
• Training-free strategy improves problem solving accuracy