Addressing Over-Refusal in LLMs with Competing Rewards

2026-06-30

AI Summary

This work addresses the issue of over-refusal in large language models during safety alignment training, where harmless user requests are erroneously rejected. To mitigate this, the authors propose SEAR, a novel framework that introduces dual roles—“reasoning player” and “response player”—within a single chain-of-thought process. The reasoning player proactively explores potentially harmful reasoning paths to generate adversarial signals, while the response player produces safe and compliant outputs. Leveraging reinforcement learning with dense process-level rewards, SEAR jointly optimizes both roles within a unified architecture, enabling precise discrimination between harmful and benign inputs. Experimental results demonstrate that SEAR significantly alleviates over-refusal and enhances robustness against attacks that directly manipulate the model’s reasoning process.

Abstract

Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones. Though this trade-off can be mitigated by training models with reinforcement learning (RL) to reason before answering, it does not remove the underlying problem that reasoning can often be a "rubber stamp" for a predetermined response. In this paper, we address the safety-refusal trade-off by rethinking how models are trained to reason about safety. Our key insight is that unsafe reasoning can itself serve as a useful exploratory signal. Rather than preemptively blocking harmful thoughts, we encourage the model to sufficiently explore unsafe reasoning but produce a safe response. The harmful exploration improves the model's ability to distinguish harmful from harmless prompts by resolving ambiguity, allowing it to remain safe while complying only when appropriate. We cast this as an adversarial optimization problem in which a reasoning player explores strategies for producing an unsafe response and an answer player ensures that the final output is safe. We train a single model with dense rewards to play both roles within one chain-of-thought, across different segments. To achieve this, we find that process rewards are crucial for stable optimization of competing objectives. Our resulting model SEAR deliberately engages in harmful reasoning as exploration while reliably flipping back to a safe answer. We demonstrate that this behavior helps mitigate over-refusal and defend against attacks that directly manipulate the reasoning to be harmful.

Problem

• over-refusal
• safety training
• language models
• harmful prompts
• refusal trade-off

Innovation

• over-refusal
• competing rewards
• reasoning exploration
• adversarial optimization
• process rewards