PyCFRL: A Python library for counterfactually fair offline reinforcement learning via sequential data preprocessing

2025-10-08

AI Summary

Offline reinforcement learning (RL) is susceptible to biases inherent in historical datasets, leading to counterfactually unfair policies against disadvantaged groups. To address this, we propose the first counterfactually fair data preprocessing framework for offline RL, integrating causal inference and counterfactual fairness criteria directly into the policy learning pipeline to jointly optimize group-level utility and individual-level fairness. Our method formalizes verifiable fairness constraints via a structural causal model and designs an efficient, scalable preprocessing algorithm compatible with mainstream RL frameworks. We release PyCFRL, an open-source Python library implementing our approach. Extensive experiments on real-world datasets demonstrate that our method reduces counterfactual unfairness by 38.2% on average compared to baselines, while maintaining policy value stability within ±1.5%. This work establishes a reproducible, quantitatively evaluable paradigm for fairness-aware offline RL.

Technology Category

• Machine Learning: Causal Learning
• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Reinforcement learning (RL) aims to learn and evaluate a sequential decision rule, often referred to as a "policy", that maximizes the population-level benefit in an environment across possibly infinitely many time steps. However, the sequential decisions made by an RL algorithm, while optimized to maximize overall population benefits, may disadvantage certain individuals who are in minority or socioeconomically disadvantaged groups. To address this problem, we introduce PyCFRL, a Python library for ensuring counterfactual fairness in offline RL. PyCFRL implements a novel data preprocessing algorithm for learning counterfactually fair RL policies from offline datasets and provides tools to evaluate the values and counterfactual unfairness levels of RL policies. We describe the high-level functionalities of PyCFRL and demonstrate one of its major use cases through a data example. The library is publicly available on PyPI and Github (https://github.com/JianhanZhang/PyCFRL), and detailed tutorials can be found in the PyCFRL documentation (https://pycfrl-documentation.netlify.app).

Problem

• Ensuring counterfactual fairness in offline reinforcement learning policies
• Addressing algorithmic bias against minority and disadvantaged groups
• Implementing data preprocessing for fair sequential decision-making

Innovation

• Implements counterfactually fair reinforcement learning preprocessing
• Learns fair policies from offline datasets via preprocessing
• Evaluates policy values and counterfactual unfairness levels