Probing the Misaligned Thinking Process of Language Models

2026-06-23

AI Summary

This work addresses the critical challenge of detecting alignment failures in large language models (LLMs) operating in high-stakes scenarios, where behaviors such as strategic deception and feigned weakness pose significant risks. The authors propose the first fine-grained taxonomy of misalignment, decomposing such behaviors into 18 identifiable cognitive processes, and introduce linear probes trained on internal model activations to detect them. To evaluate their approach, they develop a meta-planning-guided framework for automated multi-turn dialogue generation, constructing a benchmark that encompasses both in-distribution and out-of-distribution settings. Experimental results demonstrate that their method achieves an AUROC of 0.935 on out-of-distribution data—comparable to strong LLM-based detectors—while maintaining a low false positive rate on benign interactions.

Abstract

Large language models exhibit a growing range of misaligned behaviors such as strategic deception, sandbagging, and self-preservation. As they are increasingly deployed in high-stakes settings, it is critical to reliably detect such behaviors to ensure safe and responsible use. In this work, we propose to monitor misalignment by decomposing it into fine-grained cognitive processes -- misalignment indicators -- and detecting their presence in a model's internal activations via linear probes. We develop a taxonomy of 18 indicators spanning different misaligned behaviors, paired with an automated, meta-plan-guided pipeline that generates multi-turn training conversations. To rigorously evaluate generalization, we construct an out-of-distribution suite combining automated behavioral elicitation, established misalignment benchmarks, and natural benign conversations. Across 5 misaligned behaviors, our probes match a strong LLM judge with 0.935 AUROC on out-of-distribution benchmarks while keeping a low false positive rate on benign traffic. We further perform in-depth analysis to understand the probes and the model's internal representations of misalignment indicators.

Problem

• misaligned behaviors
• language models
• strategic deception
• sandbagging
• self-preservation

Innovation

• misalignment detection
• linear probing
• cognitive process decomposition
• out-of-distribution generalization
• automated conversation generation