The Illusion of Improvement: Reject Inference Strategies in Credit Scoring

2026-06-16

AI Summary

This study addresses the problem of selection bias in credit scoring caused by reject inference, which distorts model evaluation and degrades the quality of lending decisions. The authors systematically analyze existing reject inference methods, revealing a structural flaw that induces divergence between accuracy and recall during retraining cycles, and demonstrate how conventional evaluation metrics become misleading under selection bias. To overcome these limitations, they propose a controlled exploration strategy that requires no statistical assumptions and actively approves a small fraction (2–5%) of previously rejected applicants to break the feedback loop. Integrating machine learning models with this exploration mechanism, the approach is validated on three real-world datasets and two algorithms, demonstrating both effectiveness and low implementation cost, thereby offering a novel paradigm for reliable iterative improvement of credit scoring systems.

Abstract

Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality -- the ability to correctly screen out defaulters -- is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality give opposite recommendations on whether to explore: accuracy favors no exploration, while rejection quality improves with it, confirming that standard evaluation metrics are misleading under selection bias. Even minimal exploration rates (2--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.

Problem

• reject inference
• survival bias
• credit scoring
• selection bias
• rejection quality

Innovation

• reject inference
• survival bias
• controlled exploration
• credit scoring
• feedback loop