Reproducing FACTER: Fairness via Conformal Thresholding and Prompt Repair

2026-06-26

AI Summary

This study addresses the joint optimization of fairness and statistical coverage in large language model–based recommender systems by reproducing and rigorously evaluating the FACTOR framework. Leveraging conformal prediction, dynamic threshold adjustment, and iterative prompt repair, the approach achieves model-agnostic fair recommendation in both open-ended generation and constrained reranking tasks. The analysis reveals that ambiguities in the original work’s target-set evaluation led to utility discrepancies; to disentangle the contribution of prompt repair mechanisms, the authors introduce a static fairness zero-shot baseline and validate fairness efficacy under fixed candidate sets. Experimental results demonstrate that FACTOR substantially reduces violations under adaptive thresholds, yet exhibits inconsistent performance under fixed thresholds and global fairness metrics. Notably, static fairness instructions achieve semantic fairness comparable to dynamic repair methods.

Abstract

Fayyazi et al. (2025) recently proposed FACTER, a model-agnostic framework designed to jointly enforce fairness and statistical coverage in LLM-based recommendation through conformal thresholding and iterative prompt repair. In this work, we conduct a reproducibility study of the FACTER framework across diverse architectures and dataset sparsity levels, evaluating both the original open-ended generation task and a constrained re-ranking extension. Under the strict reproduction, we observe a divergence in recommendation utility, which we trace to underspecified target-set evaluation in the original study. We then use the constrained re-ranking setting to evaluate FACTER when the candidate set is fixed, and introduce a static Fair Zero-Shot baseline to isolate the contribution of the iterative prompt repair loop. Our analysis shows that FACTER consistently reduces adaptive-threshold violation counts, but that these reductions are not consistently reflected under the fixed threshold or in global fairness metrics. In the constrained ranking setting, static fairness instructions achieve comparable semantic-parity outcomes to FACTER's dynamic repair loop, suggesting that the additional online repair mechanism provides limited benefit in this formulation. All code and reproduction artifacts are available at https://github.com/oscar-omlf/facter-repr.

Problem

• reproducibility
• fairness
• conformal thresholding
• LLM-based recommendation
• prompt repair

Innovation

• conformal thresholding
• prompt repair
• fairness in LLMs
• reproducibility study
• constrained re-ranking