Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

2026-05-25

AI Summary

This study addresses the lack of uncertainty quantification and confidence calibration in the natural language outputs of activation steering, which undermines the reliability of interpretability methods. It presents the first systematic investigation of this issue, evaluating six confidence estimation techniques—including bootstrapped mode frequency and answer-token log-probabilities—across diverse prompt templates and the Qwen3 series of large language models. Using a 6,000-sample test set, the experiments demonstrate that bootstrapped mode frequency substantially outperforms conventional log-probability baselines, reducing expected calibration error (ECE) to 5.7% for Qwen3-8B and 10.3% for Qwen3.6-27B. The work further introduces a low-cost, rapid screening strategy that offers a practical solution for generating reliable activation-based explanations.

Abstract

Activation oracles aim to make the activations of other models legible to humans and yield promising results compared to white-box interpretability techniques. However, uncertainty quantification (UQ) for the natural-language outputs of such activation oracles is so far understudied. Here, we investigate 6 different methods for estimating the confidence of activation oracles and evaluate how well-calibrated their confidence scores are. Our experiments on 6,000 samples per oracle (varying verbalizer and context prompts) reveal that bootstrap mode frequency is the best-calibrated method among those tested (ECE 5.7% vs. 25.5% for the answer-word log-probability on Qwen3-8B; 10.3% vs. 13.1% on Qwen3.6-27B), and that the log-prob baseline can serve as a fast triage signal at a fraction of the cost. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilistic_activation_oracles.

Problem

• activation oracles
• uncertainty quantification
• confidence calibration
• interpretability
• language models

Innovation

• activation oracles
• uncertainty quantification
• confidence calibration
• bootstrap mode frequency
• interpretability