Workload Estimation for Unknown Tasks: A Survey of Machine Learning Under Distribution Shift

2024-03-20

arXiv.org

AI Summary

Real-time estimation of human cognitive load during human–machine collaboration under unknown tasks remains challenging—particularly because the informativeness of physiological signals dynamically shifts across tasks, violating the standard independent and identically distributed (IID) assumption. Method: This study establishes the first non-IID learning evaluation framework specifically for workload estimation. We propose a three-dimensional evaluation criterion—transferability, model complexity, and adaptability—and systematically benchmark robust distribution-shift mitigation approaches, including domain adaptation, meta-learning, online learning, invariant risk minimization, and causal inference. Contribution/Results: Empirical evaluation identifies the optimal technical pathway for dynamic, real-time environments. Our framework provides both theoretical guidance and empirical evidence for developing generalizable, subject-agnostic cognitive load estimation systems, advancing robust physiological modeling beyond the IID paradigm.

Technology Category

• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Human-robot teams involve humans and robots collaborating to achieve tasks under various environmental conditions. Successful teaming will require robots to adapt autonomously to a human teammate's internal state. An important element of such adaptation is the ability to estimate the human teammates' workload in unknown situations. Existing workload models use machine learning to model the relationships between physiological metrics and workload; however, these methods are susceptible to individual differences and are heavily influenced by other factors. These methods cannot generalize to unknown tasks, as they rely on standard machine learning approaches that assume data consists of independent and identically distributed (IID) samples. This assumption does not necessarily hold for estimating workload for new tasks. A survey of non-IID machine learning techniques is presented, where commonly used techniques are evaluated using three criteria: portability, model complexity, and adaptability. These criteria are used to argue which techniques are most applicable for estimating workload for unknown tasks in dynamic, real-time environments.

Problem

• Estimating human workload in unknown tasks using machine learning
• Overcoming generalization challenges in physiological signal-based workload models
• Evaluating ML techniques for adaptability in dynamic environments

Innovation

• Machine learning models estimate workload via physiological signals
• Techniques address generalization to unknown tasks challenges
• Criteria evaluate portability, complexity, and adaptability