Single-Query Black-Box Calibration Auditing via Logit Bias

2026-09-04

AI Summary

为解决LLM校准评估问题，通过操纵logit_bias参数，提出一种新的真校准误差估计器，实现对黑盒基础模型的有效审计。

Abstract

Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.

Problem

• Large Language Models
• calibration
• black-box
• zero-shot classifiers
• logit bias

Innovation

• logit bias
• single-query
• black-box calibration auditing
• True Calibration Error