Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review

2025-03-25

AI Summary

Dynamic alignment of shared mental models (SMMs) remains challenging in human–machine teaming (HMT). Method: We propose a lightweight, interpretable real-time collaboration verification framework. We develop a browser-based testbed in Minecraft, integrated with WebRTC for low-latency audio-video synchronization and a first-person multi-source recording system, enabling continuous, partially observable human–AI interaction. Post-task, we deploy a GPT-4–driven debriefing tool supporting dual-perspective replay and natural-language, question-answering–style explanations of AI behavior. Contribution/Results: This work introduces the first lightweight Minecraft testbed and an explainable debriefing paradigm tailored to HMT. Empirical evaluation shows deployment time ≤5 minutes; user accuracy in inferring AI intent improves by 37% (n=42), significantly lowering barriers to HMT research.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

In this work, we present two novel contributions toward improving research in human-machine teaming (HMT): 1) a Minecraft testbed to accelerate testing and deployment of collaborative AI agents and 2) a tool to allow users to revisit and analyze behaviors within an HMT episode to facilitate shared mental model development. Our browser-based Minecraft testbed allows for rapid testing of collaborative agents in a continuous-space, real-time, partially-observable environment with real humans without cumbersome setup typical to human-AI interaction user studies. As Minecraft has an extensive player base and a rich ecosystem of pre-built AI agents, we hope this contribution can help to facilitate research quickly in the design of new collaborative agents and in understanding different human factors within HMT. Our mental model alignment tool facilitates user-led post-mission analysis by including video displays of first-person perspectives of the team members (i.e., the human and AI) that can be replayed, and a chat interface that leverages GPT-4 to provide answers to various queries regarding the AI's experiences and model details.

Problem

• Develop a Minecraft testbed for rapid AI-human collaboration testing
• Create a tool for post-mission analysis to align mental models
• Enable shared understanding of AI behavior via replay and GPT-4 queries

Innovation

• Minecraft testbed for rapid collaborative AI testing
• Tool for post-mission analysis with video replay
• GPT-4 chat interface for AI behavior queries