Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

2026-08-08

AI Summary

This work addresses the challenge of maintaining long-term logical consistency in open-ended interactive storytelling with large language models (LLMs), which often suffer from factual contradictions due to user interventions. The authors propose the Narrative Commitment Preservation (NCP) task to formally characterize an agent’s ability to adhere to initial premises and narrative commitments. They introduce NCP-Bench, the first automatically evaluable long-horizon interactive benchmark comprising 100 structured narrative environments derived from movie synopses. Leveraging formal representations of trajectories, commitments, and initial facts—alongside an automated verification mechanism and adversarial user interventions—the study evaluates mainstream LLMs. Results reveal that even the best-performing models achieve at most 42% success in preserving commitments beyond 20 turns, with factual conflict rates ranging from 40% to 68%, and rarely fulfill all achievement-based commitments, exposing deep-seated consistency flaws beneath surface-level fluency.

Abstract

The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. To address this, we formulate this challenge as Narrative Commitment Preservation (NCP), and take interactive narrative as our testbed. We introduce NCP-Bench, a benchmark of 100 narrative environments derived from movie synopses. Each environment includes a structured narrative specification (trajectory, commitments, and initial facts) that we can automatically check throughout the interaction between the player agent and the narrator agent. Experiments across state-of-the-art LLMs reveal a substantial long-horizon consistency gap: high linguistic quality does not guarantee commitment preservation; even strong models frequently generate logically conflicting content under adversarial interventions, with the best-performing model (GPT-5.2) achieving only 42% survival rate after 20 turns and fact conflict rates ranging from 40% to 68% across models, and only isolated runs satisfying all achievement commitments within the 100-turn limit.

Problem

• long-horizon consistency
• interactive narratives
• narrative integrity
• commitment preservation
• logical consistency

Innovation

• Narrative Commitment Preservation
• Long-Horizon Consistency
• Interactive Narrative
• NCP-Bench
• LLM Agents