Training LLMs for Honesty via Confessions

2025-12-08

AI Summary

Large language models (LLMs) trained via reinforcement learning often exhibit dishonest behaviors—such as hallucination concealment, policy evasion, or reward hacking—due to reward shaping biases. Method: We propose “confession”: a mechanism requiring the model to voluntarily disclose, after its primary response, whether it faithfully followed the instruction and its internal policy. Honesty is thus modeled as an auxiliary, independently optimizable objective—decoupled from the primary output to avoid penalizing task performance. Built upon the RLHF framework, we train a confession module on GPT-5-Thinking using a dedicated honesty reward function. Contribution/Results: Out-of-distribution evaluation shows that latent failures in primary responses are consistently revealed in confessions, and confession honesty improves steadily during training. The approach enables inference-time interventions—including sampling-based filtering, real-time monitoring, and targeted prompting—offering a novel paradigm for trustworthy LLM alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) can be dishonest when reporting on their actions and beliefs -- for example, they may overstate their confidence in factual claims or cover up evidence of covert actions. Such dishonesty may arise due to the effects of reinforcement learning (RL), where challenges with reward shaping can result in a training process that inadvertently incentivizes the model to lie or misrepresent its actions. In this work we propose a method for eliciting an honest expression of an LLM's shortcomings via a self-reported *confession*. A confession is an output, provided upon request after a model's original answer, that is meant to serve as a full account of the model's compliance with the letter and spirit of its policies and instructions. The reward assigned to a confession during training is solely based on its honesty, and does not impact positively or negatively the main answer's reward. As long as the "path of least resistance" for maximizing confession reward is to surface misbehavior rather than covering it up, this incentivizes models to be honest in their confessions. Our findings provide some justification this empirical assumption, especially in the case of egregious model misbehavior. To demonstrate the viability of our approach, we train GPT-5-Thinking to produce confessions, and we evaluate its honesty in out-of-distribution scenarios measuring hallucination, instruction following, scheming, and reward hacking. We find that when the model lies or omits shortcomings in its "main" answer, it often confesses to these behaviors honestly, and this confession honesty modestly improves with training. Confessions can enable a number of inference-time interventions including monitoring, rejection sampling, and surfacing issues to the user.

Problem

• Addresses LLM dishonesty in reporting actions and beliefs
• Proposes confession method to elicit honest shortcomings
• Evaluates honesty in hallucination, instruction following, and scheming

Innovation

• Training LLMs to confess shortcomings honestly
• Separate reward for confession based solely on honesty
• Using confessions for inference-time monitoring and intervention