Risk-Averse Finetuning of Large Language Models

2025-01-12

AI Summary

Large language models (LLMs) occasionally generate rare yet highly harmful toxic outputs—a tail-risk phenomenon inadequately mitigated by conventional expectation-based optimization objectives. Method: This paper proposes a robust fine-tuning paradigm grounded in Conditional Value-at-Risk (CVaR), the first application of CVaR to LLM alignment training. By explicitly modeling and controlling tail risk, it overcomes the limitation of methods like RLHF that optimize only for average-case performance. The approach integrates risk-aware reinforcement learning, toxicity detection, and preference modeling to jointly optimize toxicity mitigation and generation quality. Results: On sentiment rewriting and toxicity mitigation benchmarks, the method reduces toxic output rates by up to 47%, without compromising task performance or textual quality—thereby significantly enhancing LLM safety and robustness.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts. We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events. By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.

Problem

• Large Language Model
• Harmful Content Prevention
• Creativity Preservation

Innovation

• Optimized Conditional Value at Risk
• Risk-Averse Reinforcement Learning
• Harmful Language Mitigation