AGC-Bench: Measuring Artificial General Creativity

2026-07-01

AI Summary

Current evaluations of creativity in large language models (LLMs) lack a unified benchmark and are susceptible to scoring biases. This work proposes AGC-Bench—the first large-scale, standardized, and open-source evaluation benchmark encompassing 78 creative tasks—and introduces AGC-Judge, a psychometrically grounded scoring model that calibrates LLM rating biases via Judge Response Theory. For the first time, the study identifies an AI general creativity factor, denoted as “c,” which is dissociable from general intelligence and accounts for 81.5% of cross-task variance. Empirical results demonstrate that “creativity-eliciting” prompts significantly enhance LLM performance, yet top human performers still outperform even the strongest models. The project publicly releases the benchmark, scoring model, human evaluation data, and a public leaderboard.

Abstract

Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

Problem

• Artificial General Creativity
• LLM creativity
• creativity benchmark
• domain-specific creativity
• general intelligence

Innovation

• Artificial General Creativity
• AGC-Bench
• Judge Response Theory
• creativity factor c
• AGC-Judge