Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models
This work addresses the limitation of existing conformal risk control methods, which fail to enforce risk constraints across subgroups under shifting group structures, often leading to excessive error exposure for certain populations. To bridge the gap between marginal risk guarantees and group fairness, we propose HG-CRC, a novel framework that incorporates hierarchical group structure into conformal prediction. By simultaneously imposing risk constraints at all levels of the hierarchy and employing Bonferroni correction alongside a leaf-node-prioritized threshold selection strategy, HG-CRC enables fine-grained, post-hoc calibration without model retraining. Empirical evaluation on the ARC Challenge demonstrates that our method achieves a 0% empirical violation rate and WGER = 0 for high-accuracy models, while ablation studies confirm the critical role of hierarchy depth in satisfying the prescribed risk budget.