🤖 AI Summary
This work addresses the challenge of enabling small language models to decide when to defer to human control in privacy-sensitive, offline, and cost-constrained settings while maintaining bounded risk. The authors propose a linguistically grounded confidence-based deferral mechanism and analyze, both theoretically and empirically, how calibration methods influence the trade-off between risk and coverage. Key contributions include establishing three theoretical bounds, introducing a finite-sample risk certification method based on the Clopper–Pearson interval, and correcting an answer-ranking artifact in the multiple-choice formulation of TruthfulQA. Experimental results across 22 model–task pairs show that only three achieve certified autonomy under a 20% risk budget, with none satisfying a stricter 10% threshold; furthermore, Platt scaling reduces expected calibration error (ECE) to as low as 0.02.
📝 Abstract
Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human. We study whether verbalized confidence can support risk-controlled deferral, evaluating eleven instruction-tuned models from three families, 0.5B to 14B parameters, on ARC-Challenge and TruthfulQA with 25,168 local predictions. Three theoretical results delimit what calibration can provide: strictly monotone calibration preserves the risk-coverage frontier and error-detection AUROC; temperature scaling cannot calibrate models whose confidence stays above one half while accuracy falls below it; and a Clopper-Pearson procedure converts a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption. Empirically, eight of 22 model-task pairs hit the temperature-scaling infeasibility floor within one percentage point of the predicted bound. Platt scaling reduces ECE to as low as 0.02, yet certified autonomy at a 20% risk budget is granted to only three model-task pairs and to none at 10%. We also identify and repair an answer-ordering artifact in the multiple-choice form of TruthfulQA. Calibration gives confidence semantics; certified deferral determines when small models are safe to use.