HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice

2026-06-23

AI Summary

This study systematically evaluates the ability of large language models to provide accurate and secure advice in the domain of digital privacy, security, and protection. To this end, the authors construct the first benchmark comprising 450 real-world user scenarios and develop a rule-based automated scoring system to quantitatively and qualitatively assess responses from 18 prominent models. Experimental results show that models achieve an average accuracy of 82%, yet approximately 10% of responses score below 65%, indicating a non-negligible risk of delivering incorrect or harmful guidance. This work establishes the first dedicated benchmark and scalable evaluation framework for assessing the reliability of large language models in sensitive security-critical contexts.

Abstract

This paper introduces HelpBench, a benchmark for assessing whether LLMs are capable of providing accurate help in response to questions about digital privacy, safety, and security. We curated 450 questions representing authentic user situations and developed rubrics for each question to evaluate the factual accuracy and tone of a response. Example questions touch on how to regain access to lost or suspended accounts, how to balance the trade-offs of hardware security keys versus other forms of two-factor authentication, whether a suspicious email is likely a scam, or whether an abuser might be able to track an individual based on their device peripherals. We then developed and applied an auto-rater to evaluate responses from 18 state-of-the-art LLMs. Our results indicate that while models provide high-quality advice (with scores of 82% on average), one in ten responses from models scores less than 65%, reflecting inaccurate and even harmful advice. Addressing these failures is critical for models to serve as trustworthy sources of assistance for digital privacy, safety, and security needs.

Problem

• privacy
• safety
• security
• LLMs
• advice accuracy

Innovation

• HelpBench
• LLM evaluation
• privacy advice
• security benchmark
• auto-rater