Full-range Binary Classifier Calibration for Stable Model Updates in Production

2026-07-06

AI Summary

This work addresses the challenge of prediction score distribution drift caused by frequent model retraining in security applications, which undermines downstream systems’ reliance on consistent false positive rates (FPR). To resolve this, the authors propose a novel binary classifier calibration method that operates across the full FPR spectrum, departing from conventional probability-based calibration paradigms. Their approach enables precise control over output scores at any desired FPR threshold, ensuring semantic consistency of FPR across different model versions. Built upon existing calibration primitives, the method directly optimizes FPR contracts rather than class probabilities, making it suitable for large-scale production deployment. Experiments demonstrate that within the 0.01%–10% FPR range, the relative FPR error remains below 7.2%, with a calibration artifact size under 200 KB, and the solution scales efficiently from thousands to millions of samples.

Abstract

Detection models running in adversarial environments face a malicious distribution that drifts rapidly while the benign distribution stays comparatively stable, so teams retrain and redeploy constantly to stay ahead of new threats. Retraining tends to change the output prediction scores, which breaks downstream users of the model. For these security-oriented models we need consistent false-positive rate (FPR) across all output values, whereas standard probability-calibration methods target class probability rather than an FPR contract. We introduce a method built on top of existing calibration primitives that targets the whole FPR curve, giving scores a consistent FPR meaning across deployments. On one held-out split, the observed relative FPR error was at most 2.3% from 10% down to 0.1% FPR and 7.2% at 0.01% FPR. The shipped artifact remains under 200 KB in measurements across calibration sets from 1K to 10M benign samples.

Problem

• binary classifier calibration
• false-positive rate
• model updates
• adversarial environments
• distribution drift

Innovation

• binary classifier calibration
• false positive rate (FPR)
• model stability
• adversarial environments
• production deployment