Human-in-the-Loop Pareto Optimization: Trade-off Characterization for Assist-as-Needed Training and Performance Evaluation

2026-03-24

AI Summary

This work addresses the inherent tension between task difficulty and user performance in motor skill training and rehabilitation, where systematic evaluation methods are lacking. We propose a human-in-the-loop Pareto optimization framework that, for the first time, applies Bayesian multi-objective optimization to jointly model task performance and perceived challenge level, enabling the design of personalized assistance protocols. By integrating quantitative and qualitative metrics, a haptic feedback platform, and human-in-the-loop experiments, our approach allows performance evaluation without requiring users to complete tasks independently and facilitates fair inter-individual comparisons across the full spectrum of assistance levels. User studies demonstrate the framework’s effectiveness in designing adaptive assistance-as-needed (AAN) training protocols and evaluating both group-level and individual training outcomes.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

During human motor skill training and physical rehabilitation, there is an inherent trade-off between task difficulty and user performance. Characterizing this trade-off is crucial for evaluating user performance, designing assist-as-needed (AAN) protocols, and assessing the efficacy of training protocols. In this study, we propose a novel human-in-the-loop (HiL) Pareto optimization approach to characterize the trade-off between task performance and the perceived challenge level of motor learning or rehabilitation tasks. We adapt Bayesian multi-criteria optimization to systematically and efficiently perform HiL Pareto characterizations. Our HiL optimization employs a hybrid model that measures performance with a quantitative metric, while the perceived challenge level is captured with a qualitative metric. We demonstrate the feasibility of the proposed HiL Pareto characterization through a user study. Furthermore, we present the utility of the framework through three use cases in the context of a manual skill training task with haptic feedback. First, we demonstrate how the characterized trade-off can be used to design a sample AAN training protocol for a motor learning task and to evaluate the group-level efficacy of the proposed AAN protocol relative to a baseline adaptive assistance protocol. Second, we demonstrate that individual-level comparisons of the trade-offs characterized before and after the training session enable fair evaluation of training progress under different assistance levels. This evaluation method is more general than standard performance evaluations, as it can provide insights even when users cannot perform the task without assistance. Third, we show that the characterized trade-offs also enable fair performance comparisons among different users, as they capture the best possible performance of each user under all feasible assistance levels.

Problem

• human-in-the-loop
• Pareto optimization
• assist-as-needed
• motor learning
• performance evaluation

Innovation

• Human-in-the-loop optimization
• Pareto optimization
• Assist-as-needed
• Bayesian multi-criteria optimization
• Motor skill training