Risk-Controlled Post-Processing of Decision Policies

2026-05-07

AI Summary

This work addresses the challenge of improving policy performance while adhering to user-specified risk constraints and remaining close to a given deterministic baseline policy. The authors propose a risk-controlled post-processing method that constructs a new policy maximizing fidelity to the baseline while satisfying a loss-based chance constraint. They show that the optimal solution exhibits a threshold structure, with the threshold selected via calibration data. Leveraging algorithmic stability and stochastic process theory, the method provides finite-sample guarantees for both exact risk control and near-optimality in expected performance. Empirical evaluations on COVID-19 image diagnosis, large language model routing, and multiclass classification tasks demonstrate that the approach significantly outperforms naive mixing strategies while strictly respecting the prescribed risk budget.

Abstract

Predictive models are often deployed through existing decision policies that stakeholders are reluctant to change unless a risk constraint requires intervention. We study risk-controlled post-processing: given a deterministic baseline policy, choose a new policy that maximizes agreement with the baseline subject to a chance constraint on a user-specified loss. At the population level, we show that the optimal policy has a threshold structure: it follows the baseline except on contexts where switching to the oracle fallback policy yields a large reduction in conditional violation risk. At the finite-sample level, given a fitted fallback policy and score, we develop a post-processing algorithm that uses calibration data to select a threshold. Leveraging tools from algorithmic stability and stochastic processes, we show that under regularity conditions, in the i.i.d. setting, the expected excess risk of the post-processed policy is $O(\log n/n)$. In the special case when an exact-safe fallback policy is available, the algorithm achieves precise expected risk control under exchangeability. In this setting, we also give high-probability near-optimality guarantees on the post-processed policy. Experiments on a COVID-19 radiograph diagnosis task, an LLM routing problem, and a synthetic multiclass decision task show that targeted post-processing can meet or nearly meet risk budgets while preserving substantially more agreement with the baseline than score-blind random mixing.

Problem

• risk-controlled post-processing
• decision policies
• chance constraint
• baseline agreement
• conditional violation risk

Innovation

• risk-controlled post-processing
• chance constraint
• threshold policy
• algorithmic stability
• fallback policy