QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability

2026-05-25

AI Summary

Existing approaches struggle to objectively and automatically evaluate the creative generation capabilities of large language models, often relying on multiple-choice questions or subjective human ratings. This work proposes a multi-blank cascaded story cloze benchmark that requires models to perform open-ended text generation under explicit content constraints and inter-blank dependencies. For the first time, it integrates cascaded dependencies, explicit constraints, and the “calibrated surprise” theory to construct an information-theoretic automatic scoring mechanism. This method operates without human intervention and effectively distinguishes among outputs that violate constraints, those that are mundane, and high-quality creative generations that simultaneously satisfy constraints and exhibit calibrated surprise. Consequently, it enables objective, scalable, and fully automated assessment of creative generation ability.

Abstract

Large language models (LLMs) face a dual challenge in creative capability evaluation: existing benchmarks (e.g., Story Cloze Test, HellaSwag) measure models' discriminative ability over narrative continuation using multiple-choice recognition paradigms, rather than directly measuring creative generation capability; rubric-based scoring and LLM-as-Judge methods rely on subjective dimension assessment or natural language model outputs, and cannot provide objective, automated scoring mechanisms. This paper proposes QUIET (Quality Understanding via Interlocked Evaluation Testing), a diagnostic benchmark for LLM creative capability based on multi-blank cascaded story cloze. QUIET sets N blanks (10-20) in a story with complete structure, with each blank accompanied by an explicit content constraint, and cascade dependency relationships between blanks -- the content filled into earlier blanks constrains the feasible solution space for later blanks. The evaluated model (or human participants) fills all blanks in open-ended generation mode; the results are scored by an information-theoretic automated scoring protocol without human grading. The scoring protocol directly operationalizes the "calibrated surprise" theoretical framework (Zou & Xu, 2026a). For each blank k, a composite score is computed: score = satisfy * (1 + lambda * surprise), where lambda = 1.0. Here, "satisfy" measures how well the blank filling satisfies the content constraint (objective logical reasoning judgment, not subjective aesthetic scoring), and "surprise" measures the degree of surprise given that the constraint is satisfied. Creative answers that do not satisfy the constraint score zero; answers that satisfy the constraint but are mediocre score low; answers that satisfy the constraint and are surprising score high.

Problem

• creative generation
• story cloze
• automated evaluation
• large language models
• objective scoring

Innovation

• multi-blank cascaded story cloze
• calibrated surprise
• automated scoring
• creative generation evaluation
• constraint-satisfaction and surprise