F.A.C.U.L.: Language-Based Interaction with AI Companions in Gaming

2025-11-17

AI Summary

In traditional FPS games, AI teammates rely on predefined command shortcuts (e.g., “attack”, “retreat”), lacking target specificity and semantic flexibility—limiting complex tactical coordination and immersion. This paper introduces the first natural language interaction system for real-time cooperative shooter games. Our method integrates a confidence-driven instruction decomposition framework, dynamic environment-aware entity retrieval, and spoken feedback to enable precise semantic alignment between colloquial player utterances (e.g., “clear the second floor”) and fine-grained AI agent behaviors. The key contribution lies in unifying NLP with probabilistic confidence modeling to support context-aware intent parsing and low-latency response. Evaluated in *Arena Breakout: Infinite*, the system achieves a 42% improvement in tactical command efficiency (task completion speed) and a +31-point increase in Net Promoter Score (NPS), demonstrating substantial gains in both operational effectiveness and player immersion over conventional radial-command paradigms.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Teamwork
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

In cooperative video games, traditional AI companions are deployed to assist players, who control them using hotkeys or command wheels to issue predefined commands such as ``attack'', ``defend'', or ``retreat''. Despite their simplicity, these methods, which lack target specificity, limit players' ability to give complex tactical instructions and hinder immersive gameplay experiences. To address this problem, we propose the FPS AI Companion who Understands Language (F.A.C.U.L.), the first real-time AI system that enables players to communicate and collaborate with AI companions using natural language. By integrating natural language processing with a confidence-based framework, F.A.C.U.L. efficiently decomposes complex commands and interprets player intent. It also employs a dynamic entity retrieval method for environmental awareness, aligning human intentions with decision-making. Unlike traditional rule-based systems, our method supports real-time language interactions, enabling players to issue complex commands such as ``clear the second floor'', ``take cover behind that tree'', or ``retreat to the river''. The system provides real-time behavioral responses and vocal feedback, ensuring seamless tactical collaboration. Using the popular FPS game extit{Arena Breakout: Infinite} as a case study, we present comparisons demonstrating the efficacy of our approach and discuss the advantages and limitations of AI companions based on real-world user feedback.

Problem

• Traditional AI companions lack target specificity in games
• Existing methods limit complex tactical instructions for players
• Current systems hinder immersive gameplay experiences with companions

Innovation

• Natural language processing for real-time AI interaction
• Confidence-based framework for command decomposition
• Dynamic entity retrieval for environmental awareness