The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models

2026-05-05

AI Summary

This study investigates whether language models can simulate philosophical conceptual analysis by iteratively generating counterexamples and refining definitions. The authors implement a dual-model framework that alternates between producing counterexamples and revising definitions over thousands of iterations across twenty philosophical concepts. Human experts collaborate with the models to jointly evaluate the validity of generated counterexamples, enabling systematic tracking of definitional evolution. This work represents the first systematic integration of philosophical conceptual analysis into the evaluation of language models, probing their capacity for higher-order reasoning and conceptual stability. Results indicate that approximately half of the model-generated counterexamples are judged invalid by human evaluators; while iterative refinement tends to lengthen definitions, it does not substantially improve their accuracy, and several concepts fail to converge toward stable formulations.

Abstract

Conceptual analysis -- proposing definitions and refining them through counterexamples -- is central to philosophical methodology. We study whether language models can perform this task through iterated analysis and repair chains: one model instance generates counterexamples to a proposed definition, another repairs the definition, and the process repeats. Across 20 concepts and thousands of counterexample-repair cycles, we find that, although many LM-generated counterexamples are judged invalid by both expert humans and an LM judge, the LM judge accepts roughly twice as many as humans do. Nonetheless, per-item validity judgments are moderately consistent across humans and between humans and the LM. We further find that extended iteration produces increasingly verbose definitions without improving accuracy. We also see that some concepts resist stable definitions in general. These findings suggest that while LMs can engage in philosophical reasoning, the counterexample-repair loop hits diminishing returns quickly and could be a fruitful test case for evaluating whether LMs can sustain high-level iterated philosophical reasoning.

Problem

• conceptual analysis
• counterexample
• language models
• iterated reasoning
• definition repair

Innovation

• counterexample game
• iterated conceptual analysis
• language models
• philosophical reasoning
• definition repair