CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models

2026-06-18

AI Summary

This work investigates the reasoning capabilities of large language models on combinatorial counting tasks, revealing systematic vulnerabilities in handling ordered objects, indistinguishable elements, relative positional constraints, and nested dependencies. To this end, we introduce CombEval—a dynamic benchmark grounded in typed Cofola specifications that generates natural language combinatorial problems with exact answers, enabling fine-grained control over object types, entity scale, constraint complexity, and reasoning depth. CombEval is the first framework to support scalable and controllable automatic generation and evaluation of combinatorial counting problems, integrating both direct reasoning and code-augmented paradigms while verifying solutions via a constraint solver. Experiments across eleven prominent models demonstrate consistently poor performance, with primary errors stemming from misinterpretation of constraints and incorrect application of fundamental counting principles.

Abstract

We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed Cofola specification over entities, combinatorial objects, object dependencies, and constraints, enabling controlled generation of natural-language counting problems with exact solver-verified answers. Unlike static collections, CombEval supports systematic variation of object type, entity scale, constraint count, and reasoning depth. We evaluate 11 LLMs under direct and code-augmented settings and find that models remain brittle on ordered objects, indistinguishable elements, relatively positional constraints, and nested object dependencies. Error analysis further identifies failures in constraint interpretation and counting principles. CombEval provides a diagnostic testbed for studying when and why LLMs fail at combinatorial reasoning. The code and generated benchmark suites are publicly available at \url{https://github.com/YuxuZhou-CN/combination-problem-generation}.

Problem

• combinatorial counting
• large language models
• benchmark evaluation
• reasoning failure
• constraint interpretation

Innovation

• combinatorial counting
• dynamic benchmark
• Cofola specification
• constraint reasoning
• large language models