Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes

2026-06-25

AI Summary

Current safety classifiers for large language models often underperform on complex or ambiguous prompts due to the lack of explicit modeling of user intent. This work proposes treating user intent as an explicit supervisory signal between prompts and harm labels, introducing AIMS—a dataset comprising 1,724 challenging samples annotated with corresponding intent descriptions—and systematically demonstrating the consistent benefits of intent-aware training. We develop a generalized reinforcement preference optimization (GRPO) method that uses intent fidelity as a reward signal and integrate it with supervised fine-tuning (SFT), direct preference optimization (DPO), and reasoning distillation in a multi-paradigm training framework. Experiments show that intent-guided approaches significantly enhance safety classification performance across diverse methods; GRPO achieves the best average results on five external benchmarks and attains the optimal Pareto trade-off between inference latency and F1 score.

Abstract

We argue that safety classifiers should model user intent as an explicit signal between the prompt and the final label. To study this, we introduce AIMS, a human-annotated dataset of 1,724 difficult safety prompts, each paired with an intent description and harm label. We use AIMS to evaluate intent-aware training across supervised fine-tuning, preference learning, reasoning distillation, and reinforcement learning. Despite its size, AIMS enables competitive safety classifiers across training regimes: DPO from model-generated intent errors improves over SFT, and intent-conditioned distillation outperforms reasoning-only distillation in most teacher-student pairs. Most notably, directly rewarding intent faithfulness with GRPO yields the strongest average performance across five external safety benchmarks, while our intent-aware models form the inference latency-F1 Pareto frontier. These results show that faithful intent modeling is a compact, high-quality supervision signal for more robust safety classifiers.

Problem

• safety classification
• user intent
• large language models
• harm detection
• intent modeling

Innovation

• intent-aware training
• safety classification
• AIMS dataset
• GRPO
• reasoning distillation