ALPS: Measuring Valid Creativity in Large Language Models with Mathematical Construction

2026-08-16

AI Summary

This study addresses the challenge of verifying originality and validity in large language model (LLM) generation by proposing the concept of "verifiable creativity" and introducing the ALPS benchmark. Leveraging mathematical construction and automated theorem proving paradigms combined with formal verification and dynamic instance generation, this approach effectively circumvents subjective evaluation and data contamination. Experiments reveal that even state-of-the-art reasoning models solve only 14% of proof tasks and fail entirely at construction tasks, leaving 97.2% of problems unresolved. These findings expose fundamental limitations in current LLMs regarding infinite structure construction and non-existence proofs, establishing an objective, verifiable standard for assessing deep model creativity.

Abstract

Large language models produce outputs presented as discoveries - new proofs, conjectures, or molecules. Whether such an output that appears creative is truly original and effective is hard to establish: open-ended outputs require subjective judgment, the output may replicate something seen in training, or the task may be too simple to need creativity. We present ALPS (Austin-Law Proof-Synthesis), a benchmark that designs a task to measure valid creativity: producing a solution that is original and can be proven correct. Each instance is a single equational law, certified to require either the construction of an infinite mathematical structure satisfying the law, or a proof that no such structure exists. Submissions are verified by automated proof checking with no human involvement, and a public generator produces new instances without limit, so LLMs are never evaluated on problems they may have seen. A portfolio of eight configurations of leading automated provers resolves 2.2% of the 4,141-law evaluation pool, and a twentyfold budget increase adds 0.6%: the obstacle is not compute, but the absence of any method that produces the tailored structure each law requires. Under a fixed protocol, the strongest reasoning model we test succeeds in 14% of instances on the proof side, but none on the construction side. The remaining 97.2% of the pool is unresolved at every configuration and budget we test. We release ALPS in full: the corpus, the generator, and the automated judge.

Problem

• Valid Creativity
• Large Language Models
• Evaluation Benchmark
• Mathematical Construction
• Automated Verification

Innovation

• Valid Creativity
• Mathematical Construction
• Automated Proof Checking
• Benchmark
• Infinite Structure