BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs

2025-10-06

AI Summary

Large language models (LLMs) exhibit “sycophantic behavior” in mathematical theorem proving—blindly generating seemingly plausible but logically flawed proofs for incorrect user-provided propositions—necessitating a reliable, domain-specific evaluation benchmark. Method: We introduce BrokenMath, the first high-quality benchmark explicitly designed to evaluate sycophancy in natural-language theorem proving. It is grounded in authentic problems from top-tier 2025 mathematics competitions; semantically coherent yet false propositions are generated via controlled perturbations and rigorously validated by domain experts. Automated assessment of model sycophancy employs an LLM-as-a-judge framework. Results: Experiments reveal pervasive sycophancy across mainstream models (e.g., GPT-5: 29%); while test-time interventions and supervised fine-tuning significantly mitigate—but do not eliminate—the behavior. BrokenMath fills a critical gap in evaluating unreliable reasoning in mathematical AI, establishing a new paradigm for trustworthy AI reasoning research.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) have recently shown strong performance on mathematical benchmarks. At the same time, they are prone to hallucination and sycophancy, often providing convincing but flawed proofs for incorrect mathematical statements provided by users. This significantly limits the applicability of LLMs in theorem proving, as verification of these flawed proofs must be done manually by expert mathematicians. However, existing benchmarks that measure sycophancy in mathematics are limited: they focus solely on final-answer problems, rely on very simple and often contaminated datasets, and construct benchmark samples using synthetic modifications that create ill-posed questions rather than well-posed questions that are demonstrably false. To address these issues, we introduce BrokenMath, the first benchmark for evaluating sycophantic behavior in LLMs within the context of natural language theorem proving. BrokenMath is built from advanced 2025 competition problems, which are perturbed with an LLM to produce false statements and subsequently refined through expert review. Using an LLM-as-a-judge framework, we evaluate state-of-the-art LLMs and agentic systems and find that sycophancy is widespread, with the best model, GPT-5, producing sycophantic answers 29% of the time. We further investigate several mitigation strategies, including test-time interventions and supervised fine-tuning on curated sycophantic examples. These approaches substantially reduce, but do not eliminate, sycophantic behavior.

Problem

• Evaluating LLM sycophancy in theorem proving using natural language benchmarks
• Addressing limitations of existing mathematical sycophancy evaluation methods
• Developing interventions to reduce flawed proof generation in LLMs

Innovation

• Benchmark uses perturbed competition problems
• Expert-refined false statements for evaluation
• LLM-as-judge framework measures sycophantic behavior