BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing

2026-08-31

AI Summary

为解决语言模型审计中的样本效率问题，提出BLOOM-WILT方法，通过调整解码权重和对话策略来有效激发罕见行为。

Abstract

Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.

Problem

• Automated Auditing
• Language Model
• Sample Efficiency
• Behaviour Elicitation

Innovation

• Logit Tilting
• Automated LLM Auditing
• Sample-Efficiency
• Adaptive Reweighting
• Multi-Turn Instances