A Data-Centric Approach for Safe and Secure Large Language Models against Threatening and Toxic Content

2025-04-19

AI Summary

To address toxicity, bias, and jailbreaking risks inherent in large language model (LLM) outputs, this paper proposes BART-Corrective—a data-driven, model-agnostic post-hoc toxicity correction framework. Unlike approaches requiring architectural modifications, prompt engineering, or fine-tuning, BART-Corrective employs a lightweight, pre-trained BART-based correction model to purify LLM generations in real time. It introduces a “data-centric” post-processing paradigm, integrating multi-dimensional toxicity detection (e.g., Perspective API) and enabling plug-and-play compatibility across diverse LLMs. Extensive evaluation on four major LLMs—GPT-4, PaLM2, Mistral, and Gemma—demonstrates an average 28% reduction in toxicity and up to a 23% decrease in jailbreak success rate. The framework significantly enhances safety, generalizability, and deployment flexibility without compromising generation quality or inference efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLM) have made remarkable progress, but concerns about potential biases and harmful content persist. To address these apprehensions, we introduce a practical solution for ensuring LLM's safe and ethical use. Our novel approach focuses on a post-generation correction mechanism, the BART-Corrective Model, which adjusts generated content to ensure safety and security. Unlike relying solely on model fine-tuning or prompt engineering, our method provides a robust data-centric alternative for mitigating harmful content. We demonstrate the effectiveness of our approach through experiments on multiple toxic datasets, which show a significant reduction in mean toxicity and jail-breaking scores after integration. Specifically, our results show a reduction of 15% and 21% in mean toxicity and jail-breaking scores with GPT-4, a substantial reduction of 28% and 5% with PaLM2, a reduction of approximately 26% and 23% with Mistral-7B, and a reduction of 11.1% and 19% with Gemma-2b-it. These results demonstrate the potential of our approach to improve the safety and security of LLM, making them more suitable for real-world applications.

Problem

• Mitigating harmful content in LLMs post-generation
• Reducing toxicity and jail-breaking scores effectively
• Ensuring safe and ethical use of LLMs

Innovation

• Post-generation correction mechanism for LLM safety
• BART-Corrective Model adjusts content for security
• Data-centric approach reduces toxicity and jail-breaking