Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback

2026-06-23

AI Summary

This work addresses the critical limitations of existing reinforcement learning systems—namely, their lack of safety and transparency, which often leads to undesirable behaviors. To this end, we propose the first end-to-end evaluation framework that integrates Explainable AI (XAI) with Reinforcement Learning from Human Feedback (RLHF). Supporting over 200 environments, our user-friendly, cloud-native platform enables automatic scaling and parallel management of multiple experiments. It efficiently collects large-scale human preference data, yielding reward models whose performance matches or even surpasses that of ground-truth environment rewards. Moreover, the framework reliably sustains continuous experiments involving thousands of users on standard commercial servers, significantly enhancing the interpretability, alignment, and scalability of reinforcement learning systems.

Abstract

Training safe Reinforcement Learning (RL) systems is inherently challenging, with no guarantee of avoiding unwanted behaviors. The most effective defenses against this are (i) transparency through explainability and (ii) alignment via human feedback. While both show promising results, no publicly available framework currently combines them. To address this, we introduce Themis, an XAI-enabled testing and evaluation framework for Reinforcement Learning from Human Feedback. Themis supports over 200 widely used environments and is easily configurable for experiments in RL, transparency, and alignment. Our results show that Themis can train reward models that match or outperform the environment's true reward signal using human preferences. We also provide a cloud-based platform for collecting human feedback and managing experiments. It is user-friendly, auto-scalable, and supports large participant groups across multiple experiments without extra development overhead. Tests show Themis can support one thousand users in back-to-back experiments on a modest commercial machine.

Problem

• Reinforcement Learning
• Explainable AI
• Human Feedback
• AI Safety
• Alignment

Innovation

• Explainable AI
• Reinforcement Learning from Human Feedback
• Reward Modeling
• Human-in-the-loop
• Scalable Evaluation Framework