Simulating Human Audiovisual Search Behavior

2026-02-02

AI Summary

This study addresses the challenge of coordinating perception and action under uncertainty to efficiently perform audiovisual cue-guided target search. It proposes the Sensonaut model, which, for the first time, formalizes the joint optimization of human motor behavior and sensory strategies as a resource-rational decision-making problem within a partially observable Markov decision process (POMDP) framework, explicitly capturing the trade-offs among time, effort, and accuracy. Behavioral experiments validate that the model successfully reproduces human search times, effort allocation, and characteristic error patterns across varying task complexities. The findings reveal how humans adaptively adjust their search strategies in occluded or cluttered environments, offering a theoretical foundation and computational framework for designing audiovisual interactive interfaces with reduced cognitive load.

Technology Category

• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Locating a target based on auditory and visual cues$\unicode{x2013}$such as finding a car in a crowded parking lot or identifying a speaker in a virtual meeting$\unicode{x2013}$requires balancing effort, time, and accuracy under uncertainty. Existing models of audiovisual search often treat perception and action in isolation, overlooking how people adaptively coordinate movement and sensory strategies. We present Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their body and sensory systems in ways they believe will most efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints. Our model formulates this as a resource-rational decision-making problem under partial observability. We validate the model against newly collected human data, showing that it reproduces both adaptive scaling of search time and effort under task complexity, occlusion, and distraction, and characteristic human errors. Our simulation of human-like resource-rational search informs the design of audiovisual interfaces that minimize search cost and cognitive load.

Problem

• audiovisual search
• embodied cognition
• resource-rational decision-making
• perceptual uncertainty
• human search behavior

Innovation

• embodied audiovisual search
• resource-rational decision-making
• partial observability
• adaptive sensory-motor coordination
• computational modeling