FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning

2025-05-12

AI Summary

Large language models (LLMs) frequently exhibit excessive refusal of benign queries during safety alignment, compromising practical utility. Method: To address this, we introduce the first structured dataset—comprising 16K ostensibly harmful yet actually benign queries—spanning 44 sensitive categories. We propose a graph-guided adversarial multi-agent prompting framework that integrates explicit structured reasoning with dual-path training (instruction fine-tuning and inference-time enhancement), and release a human-annotated evaluation benchmark. Contribution/Results: Evaluated across 29 state-of-the-art models, our approach significantly reduces over-refusal rates while preserving both safety performance and general capabilities. This work delivers a reproducible benchmark, a novel training paradigm, and critical open resources—advancing the co-optimization of safety and utility in LLMs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.

Problem

• Mitigating over-refusal of benign queries in LLMs
• Improving contextual safety via structured reasoning
• Reducing unnecessary refusals without compromising safety

Innovation

• Graph-informed adversarial multi-agent interaction framework
• Structured responses with explicit reasoning
• Tailored training datasets for diverse models