SGuard-v1: Safety Guardrail for Large Language Models

2025-11-16

AI Summary

To address the need for lightweight, real-time detection of multilingual harmful content and adversarial prompts (e.g., jailbreaking attacks) in LLM-based human–AI dialogue, this paper proposes a dual-model collaborative safety framework: one model specializes in multilingual harmful content identification, while the other focuses on jailbreak attack detection. Leveraging Granite-3.3-2B-Instruct (2B parameters), we perform instruction tuning using 1.4 million high-quality curated and synthetic samples, augmented with diverse adversarial prompt sets. Our approach integrates the MLCommons risk taxonomy and a fine-grained curriculum learning strategy. The method achieves state-of-the-art performance on both public and private safety benchmarks, delivering high detection accuracy, interpretable fine-grained risk classification, and calibrated confidence scores. It incurs low inference overhead and is deployment-friendly. The code and models are open-sourced under the Apache-2.0 license.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present SGuard-v1, a lightweight safety guardrail for Large Language Models (LLMs), which comprises two specialized models to detect harmful content and screen adversarial prompts in human-AI conversational settings. The first component, ContentFilter, is trained to identify safety risks in LLM prompts and responses in accordance with the MLCommons hazard taxonomy, a comprehensive framework for trust and safety assessment of AI. The second component, JailbreakFilter, is trained with a carefully designed curriculum over integrated datasets and findings from prior work on adversarial prompting, covering 60 major attack types while mitigating false-unsafe classification. SGuard-v1 is built on the 2B-parameter Granite-3.3-2B-Instruct model that supports 12 languages. We curate approximately 1.4 million training instances from both collected and synthesized data and perform instruction tuning on the base model, distributing the curated data across the two component according to their designated functions. Through extensive evaluation on public and proprietary safety benchmarks, SGuard-v1 achieves state-of-the-art safety performance while remaining lightweight, thereby reducing deployment overhead. SGuard-v1 also improves interpretability for downstream use by providing multi-class safety predictions and their binary confidence scores. We release the SGuard-v1 under the Apache-2.0 License to enable further research and practical deployment in AI safety.

Problem

• Detect harmful content in LLM prompts and responses
• Screen adversarial prompts covering 60 attack types
• Provide lightweight safety guardrails for multilingual AI deployment

Innovation

• Lightweight dual-model safety guardrail for LLMs
• ContentFilter trained on MLCommons hazard taxonomy
• JailbreakFilter covers 60 adversarial attack types