Two Empirical Studies on Audiovisual Semiotics of Uncertainty

2025-05-20

AI Summary

This study addresses the challenge of effectively representing probabilistic uncertainty through audiovisual synesthetic mappings to enhance user perception. Method: Two preregistered crowdsourced experiments were conducted, integrating reaction-time measurements, preference ratings, qualitative strategy coding, and probabilistic modeling to systematically evaluate the efficacy of multiple audiovisual dimension pairings—including loudness–area, tempo–blur, and pitch–brightness—in conveying uncertainty. Contribution/Results: We report the first empirical evidence of a significant misalignment between high-preference pairings (e.g., pitch–brightness) and semantically appropriate uncertainty mappings. Based on this finding, we propose a dual-criteria design principle—“low reaction time + high preference”—to identify optimal audiovisual mappings. This principle enables the identification of high-efficiency pairings and informs the development of a reusable, evidence-based design guideline for audiovisual uncertainty visualization.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Web data visualization
• Security and Privacy: Data transparency and provenance

Abstract

There exists limited theoretical guidance on integrating visualization and sonification. In this paper, we address this gap by investigating audiovisual semiotics for uncertainty representation: joining uncertainty visualization and sonification to combine audiovisual channels for enhancing users' perception of uncertainty. We conducted two preregistered crowd-sourced user studies. First, we assessed suitable audio/visual pairs. Then, we investigated audiovisual mappings of uncertainty. Here, we use probability as it is an easily communicated aspect of uncertainty. We analyzed the participants' preferences and reaction times in both user studies. Additionally, we explored the strategies employed by participants through qualitative analysis. Our results reveal audiovisual mappings that lead to particularly strong preferences and low reaction times. Furthermore, we found that preferred audio/visual pairs are not necessarily suitable audiovisual mappings of uncertainty. For example, while pitch paired with brightness was preferred as a pair, it was not well suited as a mapping for uncertainty. We recommend audiovisual mappings of uncertainty that lead to low reaction times and high preferences in both user studies. This paper presents guidelines to anyone seeking to employ audiovisual representations for uncertainty, contributing to enhancing the perception of uncertainty.

Problem

• Investigates audiovisual semiotics for uncertainty representation
• Assesses suitable audio/visual pairs for uncertainty mapping
• Recommends effective audiovisual mappings to enhance uncertainty perception

Innovation

• Combining visualization and sonification for uncertainty representation
• Assessing audio/visual pairs through crowd-sourced user studies
• Recommending audiovisual mappings based on preferences and reaction times