LLM-Mediated Human-AI Interaction in Search and Rescue: Impact of Expertise on Attentional Allocation

2026-06-17

AI Summary

This study investigates how large language models (LLMs) influence attention allocation, decision-making behavior, and situational awareness among individuals with varying expertise during search-and-rescue tasks. Using a simulated environment, the research integrates eye-tracking data, behavioral logs, and task performance metrics to compare expert and novice performance with and without LLM guidance. Results indicate that while LLM assistance improves unit efficiency—yielding higher rewards and more rescues per action—it does not increase total rescues. Eye-tracking reveals a shift in user attention toward the chat interface; experts maintain environmental scanning and actively cross-verify AI suggestions, whereas novices tend to rely passively on model outputs. The study proposes a “verification loop” mechanism, underscoring the critical role of real-time alignment between AI recommendations and environmental states in preserving situational awareness during human-AI collaboration.

Abstract

Human-AI teaming (HAT) increasingly involves AI systems that provide real-time, context-aware guidance in complex tasks. While such systems can improve performance, their effectiveness depends on how they shape human cognition and behavior. In particular, AI assistance can introduce cognitive demands and influence attention, planning, and interaction with the task environment, with effects that can vary across levels of expertise. This work investigates these mechanisms in a simulated search and rescue (SAR) environment. We compare human performance under two LLM (Large Language Model)-guided conditions and a no-LLM baseline, and analyze interaction at multiple levels, including task performance, eye-tracking measures, and planning behavior. Eye tracking provides fine-grained insight into attention allocation and interaction with AI guidance, while behavioral measures capture how users structure and adapt their decisions over time. Results indicate that LLM guidance enhanced task efficiency (higher rewards and victims-per-step) but did not increase total victims saved. Eye-tracking data revealed an attention-guidance trade-off, with visual resources shifting to the chat interface alongside increased pupil size variability. Expertise moderated this effect: novices exhibited passive AI reliance, whereas experts maintained a "verification loop" through persistent environmental scanning. These findings suggest that LLM-mediated teaming efficacy depends on the operator's ability to cross-reference AI guidance with ground truth to maintain situational awareness.

Problem

• Human-AI teaming
• Large Language Model
• attentional allocation
• search and rescue
• expertise

Innovation

• LLM-mediated teaming
• attentional allocation
• eye-tracking
• expertise modulation
• human-AI interaction