Evaluating In Silico Creativity: An Expert Review of AI Chess Compositions

2025-10-27

AI Summary

Existing generative AI systems lack aesthetic judgment and artistic validation in chess problem composition. Method: This study proposes a collaborative generation framework integrating legality constraints, aesthetic heuristic rules, and human expert evaluation—introducing, for the first time, an authoritative review mechanism composed of multiple internationally recognized chess composers to establish a human-centered paradigm for evaluating AI creativity. Contribution/Results: Experimental evaluation demonstrates that the generated problems achieve high ratings from three grandmaster composers across novelty, counterintuitiveness, and artistic expressiveness; several compositions meet professional human-composer standards. This work not only validates generative AI’s potential in structured artistic creation but also pioneers an interpretable, empirically assessable methodology for chess aesthetics generation—bridging computational reasoning with domain-specific artistic criteria.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The rapid advancement of Generative AI has raised significant questions regarding its ability to produce creative and novel outputs. Our recent work investigates this question within the domain of chess puzzles and presents an AI system designed to generate puzzles characterized by aesthetic appeal, novelty, counter-intuitive and unique solutions. We briefly discuss our method below and refer the reader to the technical paper for more details. To assess our system's creativity, we presented a curated booklet of AI-generated puzzles to three world-renowned experts: International Master for chess compositions Amatzia Avni, Grandmaster Jonathan Levitt, and Grandmaster Matthew Sadler. All three are noted authors on chess aesthetics and the evolving role of computers in the game. They were asked to select their favorites and explain what made them appealing, considering qualities such as their creativity, level of challenge, or aesthetic design.

Problem

• Evaluating AI creativity in chess puzzle generation
• Assessing aesthetic and novel qualities of AI compositions
• Expert validation of computer-generated chess puzzle creativity

Innovation

• AI system generates aesthetically appealing chess puzzles
• Puzzles feature novel and counter-intuitive solution methods
• Creativity evaluated by world-renowned chess composition experts