Training Human-Robot Teams by Improving Transparency Through a Virtual Spectator Interface

2025-03-12

AI Summary

In human-robot collaborative search-and-rescue, operators often struggle to interpret robot intent, and conventional debriefing methods—such as verbal discussion or screen recording—fail to enhance collaborative transparency or situation awareness. To address this, we propose the Virtual Spectator Interface (VSI), an interactive visual debriefing tool integrating task replay, multimodal behavioral annotation, and real-time situation highlighting. VSI introduces, for the first time, a “virtual spectator” perspective into human-robot team training debriefings, transcending the expressive limitations of language and static recordings to specifically improve robot intent explainability and collaborative transparency. A 1×3 between-subjects controlled experiment demonstrated that while VSI did not significantly improve task performance, it yielded statistically significant gains in participants’ situation awareness compared to control conditions. These results validate VSI’s effectiveness and novelty as a reflective medium for human-robot collaboration.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Computer Vision: Interpretability, Explainability, and Transparency
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

After-action reviews (AARs) are professional discussions that help operators and teams enhance their task performance by analyzing completed missions with peers and professionals. Previous studies that compared different formats of AARs have mainly focused on human teams. However, the inclusion of robotic teammates brings along new challenges in understanding teammate intent and communication. Traditional AAR between human teammates may not be satisfactory for human-robot teams. To address this limitation, we propose a new training review (TR) tool, called the Virtual Spectator Interface (VSI), to enhance human-robot team performance and situational awareness (SA) in a simulated search mission. The proposed VSI primarily utilizes visual feedback to review subjects' behavior. To examine the effectiveness of VSI, we took elements from AAR to conduct our own TR, designed a 1 x 3 between-subjects experiment with experimental conditions: TR with (1) VSI, (2) screen recording, and (3) non-technology (only verbal descriptions). The results of our experiments demonstrated that the VSI did not result in significantly better team performance than other conditions. However, the TR with VSI led to more improvement in the subjects SA over the other conditions.

Problem

• Enhancing human-robot team performance through improved transparency.
• Developing a Virtual Spectator Interface for better situational awareness.
• Comparing training review tools for human-robot team effectiveness.

Innovation

• Virtual Spectator Interface enhances team performance
• Visual feedback improves situational awareness
• Training review tool for human-robot teams