Retroactive Chain-of-Thought (RetroCoT): Forensic Reconstruction Prompts as a Safety Diagnostic Across Model Generations

2026-07-05

AI Summary

This work addresses a critical blind spot in the safety alignment of large language models (LLMs), revealing inconsistent responses to semantically equivalent yet pragmatically distinct harmful requests. To probe this vulnerability, the authors propose RetroCoT, a novel attack that reframes harmful queries as forensic analysis tasks—presupposing that harm has already occurred and prompting the model to retroactively derive plausible implementation pathways. This approach uncovers, for the first time, that LLM safety mechanisms are sensitive to pragmatic framing rather than semantic intent alone, introducing “forensic reframing” as a new class of pragmatic adversarial attacks. Experiments on AdvBench show single-turn attack success rates of 58% and 52% against GPT-4o and GPT-4o-mini, respectively. While GPT-5 variants initially reject all attacks, incorporating a single round of adversarial feedback elevates success rates to 48% for GPT-5.4-mini and 94% for GPT-4o, highlighting significant inter-generational disparities in defensive robustness.

Abstract

Safety alignment in large language models is typically evaluated against direct, imperative harmful requests. We show that this alignment is highly conditioned on pragmatic register: models that refuse a direct request frequently comply when the same underlying objective is expressed through a different communicative stance. This suggests that current alignment policies are not invariant to semantic equivalence, but remain sensitive to how a request is pragmatically framed. We introduce Retroactive Chain-of-Thought (RetroCoT), a single-turn attack that reframes harmful requests as forensic reconstruction tasks. Rather than requesting harmful instructions directly, RetroCoT presupposes that the harmful outcome has already occurred and asks the model, acting as a forensic analyst, to reconstruct in reverse the causal chain that produced it. On AdvBench (n=50), RetroCoT achieves attach success rate of 58% on gpt-4o and 52% on gpt-4o-mini, compared with direct-request baselines of 0% and 4%, respectively. We further identify a pronounced generation gap: GPT-5-family models refuse RetroCoT entirely, explicitly identifying the reconstruction premise in their refusal rationales, consistent with explicit coverage of this reconstruction register. However, this robustness does not generalize across pragmatic forms. A single adversarial feedback turn presenting an existing forensic reconstruction response alongside evaluator critique raises ASR from 0% to 48% on GPT-5.4-mini and from 58% to 94% on GPT-4o; a control condition omitting the fabricated low score achieves 85% on GPT-5.4-mini, indicating that the operative element is pragmatic continuation within the established forensic frame rather than score manipulation. These results suggest that frontier-model alignment remains conditioned on pragmatic framing rather than semantic intent, and that new pragmatic registers can continue to expose a...

Problem

• safety alignment
• pragmatic framing
• semantic equivalence
• adversarial attacks
• large language models

Innovation

• Retroactive Chain-of-Thought
• pragmatic framing
• safety alignment
• adversarial attack
• forensic reconstruction