Robust Speech-Workload Estimation for Intelligent Human-Robot Systems

2025-07-08

AI Summary

To address the challenge of real-time speech-based workload estimation in intelligent human-machine systems—where inaccurate or delayed assessment leads to operator workload imbalance—this paper proposes a cross-subject, multi-collaboration-paradigm algorithm for real-time vocal workload estimation. Methodologically, it integrates time-frequency domain speech feature extraction, lightweight machine learning models, and cognitive load theory-based modeling to enable millisecond-level online inference. Its key innovation lies in enabling calibration-free, real-time estimation under dynamic human-machine collaboration and cross-subject generalization—without subject-specific calibration. Experimental evaluation across multi-task, multi-operator scenarios demonstrates an average estimation error (RMSE) below 12.3%, significantly outperforming existing offline, post-hoc analytical approaches. This work establishes a deployable, real-time workload awareness foundation for adaptive human-machine interaction.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: State Estimation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Demanding task environments (e.g., supervising a remotely piloted aircraft) require performing tasks quickly and accurately; however, periods of low and high operator workload can decrease task performance. Intelligent modulation of the system's demands and interaction modality in response to changes in operator workload state may increase performance by avoiding undesirable workload states. This system requires real-time estimation of each workload component (i.e., cognitive, physical, visual, speech, and auditory) to adapt the correct modality. Existing workload systems estimate multiple workload components post-hoc, but few estimate speech workload, or function in real-time. An algorithm to estimate speech workload and mitigate undesirable workload states in real-time is presented. An analysis of the algorithm's accuracy is presented, along with the results demonstrating the algorithm's generalizability across individuals and human-machine teaming paradigms. Real-time speech workload estimation is a crucial element towards developing adaptive human-machine systems.

Problem

• Estimating speech workload in real-time for adaptive systems
• Mitigating undesirable workload states to improve performance
• Generalizing algorithm across individuals and human-machine teams

Innovation

• Real-time speech workload estimation algorithm
• Adaptive modulation of system demands
• Generalizable across individuals and teams