Clustering Context in Off-Policy Evaluation

2025-02-28

AI Summary

In offline policy evaluation, importance sampling (IPS) estimators suffer from severe variance inflation and bias accumulation when the behavior and target policies differ substantially. To address this, we propose a context-clustering–based information sharing framework—the first to incorporate context clustering into off-policy evaluation—enabling joint modeling across similar contexts to mitigate sparse feedback. We theoretically characterize its bias–variance trade-off and statistical convergence rate. Experiments on synthetic and real-world recommendation datasets demonstrate that our method reduces average relative estimation error by over 30% in data-scarce regimes, significantly outperforming IPS and its variants. The core innovation lies in leveraging contextual structure to enable cross-sample information transfer, thereby enhancing both robustness and accuracy of policy value estimation.

Technology Category

• Machine Learning: Online Learning & Bandits
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Off-policy evaluation can leverage logged data to estimate the effectiveness of new policies in e-commerce, search engines, media streaming services, or automatic diagnostic tools in healthcare. However, the performance of baseline off-policy estimators like IPS deteriorates when the logging policy significantly differs from the evaluation policy. Recent work proposes sharing information across similar actions to mitigate this problem. In this work, we propose an alternative estimator that shares information across similar contexts using clustering. We study the theoretical properties of the proposed estimator, characterizing its bias and variance under different conditions. We also compare the performance of the proposed estimator and existing approaches in various synthetic problems, as well as a real-world recommendation dataset. Our experimental results confirm that clustering contexts improves estimation accuracy, especially in deficient information settings.

Problem

• Improves off-policy evaluation accuracy using context clustering.
• Addresses performance degradation in baseline off-policy estimators.
• Validates estimator effectiveness in synthetic and real-world datasets.

Innovation

• Clusters contexts to improve estimation accuracy
• Shares information across similar contexts
• Theoretical analysis of bias and variance