Collaborative Storytelling and LLM: A Linguistic Analysis of Automatically-Generated Role-Playing Game Sessions

2025-03-26

AI Summary

This study investigates the genre affiliation of role-playing game (RPG) dialogues generated by large language models (LLMs) in zero-intervention settings. Methodologically, it employs quantitative linguistic analyses—including word frequency distributions, syntactic complexity metrics, and register-specific marker identification—to systematically compare LLM-generated RPG texts against three benchmark corpora: human-recorded RPG sessions, spontaneous conversational speech, and narrative fiction books. Results reveal that LLM-generated RPG discourse constitutes a distinct hybrid genre—neither prototypical spoken nor standard written language—and is significantly divergent from all comparative corpora. Crucially, this emergent genre is not attributable to superficial imitation of human interaction but rather stems from the structural embedding of heterogeneous narrative sources within the LLMs’ training data. This work provides the first systematic empirical evidence identifying and characterizing this genre, thereby uncovering a fundamental mechanism by which training corpus composition shapes LLMs’ narrative stylistic output.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Role-playing games (RPG) are games in which players interact with one another to create narratives. The role of players in the RPG is largely based on the interaction between players and their characters. This emerging form of shared narrative, primarily oral, is receiving increasing attention. In particular, many authors investigated the use of an LLM as an actor in the game. In this paper, we aim to discover to what extent the language of Large Language Models (LLMs) exhibit oral or written features when asked to generate an RPG session without human interference. We will conduct a linguistic analysis of the lexical and syntactic features of the generated texts and compare the results with analyses of conversations, transcripts of human RPG sessions, and books. We found that LLMs exhibit a pattern that is distinct from all other text categories, including oral conversations, human RPG sessions and books. Our analysis has shown how training influences the way LLMs express themselves and provides important indications of the narrative capabilities of these tools.

Problem

• Analyze oral vs written features in LLM-generated RPG sessions
• Compare LLM text with human RPG sessions and books
• Assess training impact on LLM narrative expression capabilities

Innovation

• LLM-generated RPG session linguistic analysis
• Comparison with human RPG and books
• Training influences LLM narrative expression