CHARM: Considering Human Attributes for Reinforcement Modeling

2025-06-16

AI Summary

This study investigates how inherent human teacher attributes—such as robotics experience and educational background—affect the pattern and quality of feedback provided during reinforcement learning. Through a dual-task behavioral experiment in a shared public space, 46 participants were recruited; multivariate correlation analysis and human-factor modeling were employed to systematically uncover, for the first time, statistically significant effects of demographic and behavioral features on feedback bias. Based on these findings, we propose a novel paradigm for predicting feedback value that jointly incorporates demographic information and real-time behavioral features. The resulting model achieves significantly higher accuracy in feedback quality prediction compared to baseline methods relying solely on task-level statistics. All experimental data, source code, and an open-source prediction toolkit are publicly released, providing both theoretical foundations and practical tools for trustworthy human-in-the-loop reinforcement learning.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Intelligent Robots: Human-Robot Interaction
• Machine Learning: Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Learning from Human Feedback has recently achieved significant success in various fields, and its performance is highly related to feedback quality. While much prior work acknowledged that human teachers' characteristics would affect human feedback patterns, there is little work that has closely investigated the actual effects. In this work, we designed an exploratory study investigating how human feedback patterns are associated with human characteristics. We conducted a public space study with two long horizon tasks and 46 participants. We found that feedback patterns are not only correlated with task statistics, such as rewards, but also correlated with participants' characteristics, especially robot experience and educational background. Additionally, we demonstrated that human feedback value can be more accurately predicted with human characteristics compared to only using task statistics. All human feedback and characteristics we collected, and codes for our data collection and predicting more accurate human feedback are available at https://github.com/AABL-Lab/CHARM

Problem

• Investigates how human feedback patterns relate to human characteristics
• Explores correlation between feedback and attributes like robot experience
• Improves human feedback prediction accuracy using personal attributes

Innovation

• Explores human attributes in feedback modeling
• Links feedback patterns to personal characteristics
• Predicts feedback better with human traits