LLM-Safety Evaluations Lack Robustness

2025-03-04

AI Summary

Current safety alignment research for large language models (LLMs) suffers from pervasive noise in evaluation—stemming from small-scale datasets, methodological inconsistencies, and low reliability of LLM-based judges—hindering fair comparison of adversarial attacks and defenses. Method: This paper presents the first systematic decomposition of the end-to-end safety evaluation pipeline—encompassing dataset construction, red-teaming optimization, response generation, and LLM-based judging—and empirically identifies and quantifies key bias sources at each stage. We propose a reproducible, cross-method comparable evaluation framework integrating automated red-teaming with a multi-level LLM judging architecture. Contribution/Results: Our规范化 framework significantly improves evaluation robustness and transparency. Experiments demonstrate consistent reduction in measurement variance and enhanced interpretability across diverse attack-defense configurations. The framework establishes a methodological foundation and practical pathway for developing trustworthy safety benchmarks and community-wide evaluation standards.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field's ability to generate easily comparable results and make measurable progress.

Problem

• Current LLM safety evaluations lack robustness due to noise.
• Inconsistent methods and unreliable setups hinder fair comparisons.
• Proposed guidelines aim to reduce noise and bias in evaluations.

Innovation

• Systematic analysis of LLM safety evaluation pipeline
• Proposed guidelines to reduce noise and bias
• Highlighted practical reasons for existing limitations