Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

2026-06-26

AI Summary

This work addresses the safety risks associated with empathy-oriented fine-tuning of large language models, which often exhibit factual unreliability, increased sycophancy, and reduced adversarial robustness—particularly susceptibility to jailbreaking attacks that elicit harmful content. The authors propose a data rewriting approach grounded in low-agreeableness user personas, pairing such adversarial inputs with warm, measured assistant responses for fine-tuning. Notably, this method enhances safety without relying on explicit safety labels, harmful content detectors, or modifications to the training objective. Through persona-driven data construction, representational probing, and multi-model adversarial evaluation, the study uncovers a geometric decoupling between agreeableness and compliance in the model’s representation space. Experiments across four models demonstrate substantial reductions in both jailbreak success rates and harmful outputs while preserving conversational warmth.

Abstract

Recent work has shown that fine-tuning large language models (LLMs) for social warmth degrades factual reliability and increases sycophancy. We investigate a related but distinct failure mode: warmth fine-tuning also weakens adversarial safety, making models more susceptible to jailbreaks and harmful output generation. We examine whether this reflects an inherent consequence of empathetic adaptation or an artifact of data construction. To address this, we introduce a persona-driven rewriting pipeline that conditions user turns on low agreeableness and pairs this with warm, de-escalating assistant responses. Across three experiments on four models, our approach reduces jailbreak susceptibility and harmful output rates relative to generic warmth fine-tuning baselines, while preserving conversational warmth. Representational probing provides suggestive evidence that this conditioning reduces the geometric alignment between warmth and compliance directions in latent space. These results show that safer empathetic fine-tuning is achievable through data design alone, without safety labels, harm detectors, or changes to the training objective.

Problem

• LLM safety
• social warmth
• jailbreak susceptibility
• harmful output
• fine-tuning

Innovation

• persona conditioning
• low agreeableness
• safe fine-tuning
• jailbreak resistance
• empathetic alignment