Synthetic Counterfactual Labels for Efficient Conformal Counterfactual Inference

2025-09-04

AI Summary

This paper addresses the problem of overly conservative prediction intervals for individual counterfactual outcomes under treatment imbalance and scarcity of counterfactual samples. To tackle this, we propose SP-CCI—a framework that synthesizes high-quality counterfactual labels using pretrained models, integrates risk-controlling prediction sets (RCPS), and employs importance-weighted debiasing to tighten intervals while strictly guaranteeing marginal coverage. Technically, SP-CCI unifies compliance-aware reasoning, prediction-driven inference (PPI), and approximate/exact importance-weighted calibration, with theoretical guarantees on coverage validity. Experiments across multiple benchmark datasets demonstrate that SP-CCI consistently reduces interval width—thereby enhancing both the practical utility and precision of counterfactual inference—without compromising coverage guarantees.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

This work addresses the problem of constructing reliable prediction intervals for individual counterfactual outcomes. Existing conformal counterfactual inference (CCI) methods provide marginal coverage guarantees but often produce overly conservative intervals, particularly under treatment imbalance when counterfactual samples are scarce. We introduce synthetic data-powered CCI (SP-CCI), a new framework that augments the calibration set with synthetic counterfactual labels generated by a pre-trained counterfactual model. To ensure validity, SP-CCI incorporates synthetic samples into a conformal calibration procedure based on risk-controlling prediction sets (RCPS) with a debiasing step informed by prediction-powered inference (PPI). We prove that SP-CCI achieves tighter prediction intervals while preserving marginal coverage, with theoretical guarantees under both exact and approximate importance weighting. Empirical results on different datasets confirm that SP-CCI consistently reduces interval width compared to standard CCI across all settings.

Problem

• Constructing reliable prediction intervals for counterfactual outcomes
• Addressing overly conservative intervals in conformal counterfactual inference
• Reducing interval width while preserving marginal coverage guarantees

Innovation

• Augment calibration with synthetic counterfactual labels
• Debiasing step using prediction-powered inference
• Risk-controlling prediction sets with importance weighting