Can LLMs Reliably Self-Report Adversarial Prefills, and How?

2026-06-22

AI Summary

This study investigates whether large language models (LLMs) can reliably self-identify when their outputs have been manipulated under adversarial prefilling attacks. Through systematic evaluation of ten open-source instruction-tuned models—employing linear probing, directional orthogonalization, and multiple LoRA fine-tuning strategies (SFT, GRPO, DPO)—the work reveals, for the first time, the unreliability of LLMs’ self-reporting in safety-critical scenarios: on average, 27.3% of samples erroneously claim their outputs align with their intended behavior. The research further finds that introspective signals correlate with refusal tendencies, and probe phrasing significantly influences model responses. While directional orthogonalization mitigates certain biases, fine-tuning unexpectedly widens the gap in intent recognition and increases attack success rates.

Abstract

Prior work shows that large language models (LLMs) exhibit introspective capability on benign tasks. We extend the question to safety contexts and examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack. Across ten open-weight instruction-tuned LLMs (3B to 70B) and four safety benchmarks, no model reliably recognizes its own compromised outputs, with models claiming intent on prefilled responses at an average rate of $27.3\%$. Introspective signal stems largely from safety- and refusal-related reasoning. Orthogonalizing models' weights against the refusal direction collapses the gap between claiming rates on prefilled and natural outputs to near zero, though the direction is not its unique mediator. The signal is also probe-dependent: framing the question as internal intention versus external tampering elicits qualitatively different responses on the same models. We test three LoRA finetuning methods (SFT, GRPO, DPO) on eight models from 3B to 27B; all three widen the intention-probe gap on every model from 8B to 27B, with method ranking varying by model. The intervention does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. These findings outline mechanisms underpinning the observed introspective signals in safety contexts and highlight risks in the reliability of LLM self-reports.

Problem

• adversarial prefill
• self-report
• large language models
• introspection
• safety

Innovation

• adversarial prefill
• LLM introspection
• safety alignment
• refusal direction
• LoRA finetuning