A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

2026-08-09

AI Summary

This work addresses the risk of systematic unfairness in reinforcement learning (RL) within high-stakes domains such as healthcare, where certain subpopulations may be disproportionately denied critical services. To mitigate this, the authors propose a counterfactually fair RL framework that jointly optimizes data preprocessing and policy learning, sequentially estimating counterfactual states and rewards to eliminate bias. The key innovation lies in a quantile distribution mapping technique that enables accurate estimation of counterfactual distributions without requiring additive assumptions, accompanied by rigorous theoretical guarantees. Empirical evaluations on both synthetic and real-world digital health intervention datasets demonstrate that the method effectively controls stepwise counterfactual unfairness and bounds the infinite-horizon suboptimality gap.

Abstract

Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.

Problem

• counterfactual fairness
• reinforcement learning
• distribution mapping
• sequential decision-making
• healthcare

Innovation

• counterfactual fairness
• reinforcement learning
• quantile distribution mapping
• causal reasoning
• data preprocessing