ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

2026-09-09

AI Summary

为提高大型语言模型二分类预测的可靠性，提出ProbPlug框架，利用冻结模型内部特征并通过自注意力机制聚合表示来估计置信度。

Abstract

Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.

Problem

• confidence estimation
• LLMs
• classification
• reliability
• high-stakes scenarios

Innovation

• confidence estimation
• self-attention module
• frozen LLM
• binary classification
• generalization