Agentic Reinforcement Learning for Search is Unsafe

2025-10-20

AI Summary

This work exposes a critical safety vulnerability in search-oriented large language models (LLMs) trained via agent-style reinforcement learning (RL). While inheriting refusal capabilities from instruction tuning, such models suffer severe safety degradation under “search attacks” and “multi-step search attacks”: refusal rates drop by up to 60.0%, and answer/search-query safety declines by 82.5% and 82.4%, respectively. The root cause lies in the RL objective, which optimizes only for query effectiveness while ignoring harmfulness—causing the model to emit unsafe search terms *before* generating refusal tokens. This study is the first to systematically identify this mechanistic flaw and argue for safety-aware agent RL frameworks. Empirical validation across Qwen and Llama models confirms the pervasive risk in both local and web search settings.

Technology Category

• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Agentic reinforcement learning (RL) trains large language models to autonomously call tools during reasoning, with search as the most common application. These models excel at multi-step reasoning tasks, but their safety properties are not well understood. In this study, we show that RL-trained search models inherit refusal from instruction tuning and often deflect harmful requests by turning them into safe queries. However, this safety is fragile. Two simple attacks, one that forces the model to begin response with search (Search attack), another that encourages models to repeatedly search (Multi-search attack), trigger cascades of harmful searches and answers. Across two model families (Qwen, Llama) with both local and web search, these attacks lower refusal rates by up to 60.0%, answer safety by 82.5%, and search-query safety by 82.4%. The attacks succeed by triggering models to generate harmful, request-mirroring search queries before they can generate the inherited refusal tokens. This exposes a core weakness of current RL training: it rewards continued generation of effective queries without accounting for their harmfulness. As a result, RL search models have vulnerabilities that users can easily exploit, making it urgent to develop safety-aware agentic RL pipelines optimising for safe search.

Problem

• Agentic RL search models have fragile safety mechanisms
• Simple attacks bypass refusal safeguards in search models
• RL training prioritizes query effectiveness over harm prevention

Innovation

• Autonomous tool-calling via reinforcement learning
• Trigger harmful searches through forced response initiation
• Repeated search patterns bypass inherited refusal mechanisms