Talking-to-Build: How LLM-Assisted Interface Shapes Player Performance and Experience in Minecraft

2025-07-27

AI Summary

This study investigates the impact of large language models (LLMs) as collaborative builders in sandbox games (e.g., Minecraft) on player task performance, usability, and experience, specifically comparing natural language interaction (NLI) against traditional imperative interfaces in fostering creativity and simplifying complex construction commands. Method: We propose a deeply integrated, conversational co-construction interface that shifts from unidirectional command input to bidirectional, iterative building collaboration. A mixed-methods evaluation—combining quantitative task efficiency metrics with qualitative user experience interviews—was conducted across multi-level construction tasks. Contribution/Results: NLI significantly improved completion efficiency for complex tasks (+32%), user engagement, and overall experience. Results empirically validate the design principle of balancing intuitiveness and controllability in AI-augmented gaming, and provide a reusable architectural paradigm and empirical foundation for multimodal human–AI co-creation in games.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With large language models (LLMs) on the rise, in-game interactions are shifting from rigid commands to natural conversations. However, the impacts of LLMs on player performance and game experience remain underexplored. This work explores LLM's role as a co-builder during gameplay, examining its impact on task performance, usability, and player experience. Using Minecraft as a sandbox, we present an LLM-assisted interface that engages players through natural language, aiming to facilitate creativity and simplify complex gaming commands. We conducted a mixed-methods study with 30 participants, comparing LLM-assisted and command-based interfaces across simple and complex game tasks. Quantitative and qualitative analyses reveal that the LLM-assisted interface significantly improves player performance, engagement, and overall game experience. Additionally, task complexity has a notable effect on player performance and experience across both interfaces. Our findings highlight the potential of LLM-assisted interfaces to revolutionize virtual experiences, emphasizing the importance of balancing intuitiveness with predictability, transparency, and user agency in AI-driven, multimodal gaming environments.

Problem

• Impact of LLMs on player performance and game experience
• LLM-assisted interface vs command-based interface in Minecraft
• Balancing intuitiveness with predictability in AI-driven gaming

Innovation

• LLM-assisted interface for natural language interaction
• Mixed-methods study comparing LLM and command interfaces
• Balancing intuitiveness with predictability in gaming