Safety Training Persists Through Helpfulness Optimization in LLM Agents

2026-02-13

arXiv.org

AI Summary

研究通过直接偏好优化方法在多步骤工具使用场景下提升LLM的安全性和帮助性，发现安全性训练效果在后续的帮助性训练中保持稳定，但两者存在负相关。

Abstract

Safety post-training has been studied extensively in single-step"chat"settings where safety typically refers to refusing harmful requests. We study an"agentic"(i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We compare the effects of running direct preference optimization (DPO) on safety or helpfulness alone vs both metrics sequentially. As expected, training on one metric alone results in an extreme point along this frontier. However, unlike prior work, we find that safety training persists through subsequent helpfulness training. We also find that all training configurations end up near a linear Pareto frontier with $R^2 = 0.77$. Even post-training on both metrics simultaneously simply results in another point on the frontier rather than finding a"best of both worlds"strategy, despite the presence of such strategies in our DPO dataset. Overall, our findings underscore the need for better understanding of post-training dynamics.

Problem

• safety
• helpfulness
• direct preference optimization
• agentic setting
• post-training

Innovation

• direct preference optimization
• safety training
• helpfulness optimization