CyberMaskQA: A Privacy-Aware Benchmark for Evaluating Large Language Models in Cybersecurity Question Answering

2026-05-23

AI Summary

This work addresses the challenge that current large language models struggle to simultaneously support contextual reasoning and privacy preservation in cybersecurity question answering, compounded by the absence of benchmark datasets that jointly evaluate operational reasoning capabilities and privacy retention. To bridge this gap, we introduce the first privacy-aware question-answering benchmark tailored to the cybersecurity domain, grounded in real-world organizational scenarios. The benchmark incorporates causal dependencies between assets and permissions and features fine-grained annotations of sensitive entities to enable controlled de-identification. By combining human-curated base scenarios with semantic expansions generated by large language models—and integrating entity recognition with masking mechanisms—our benchmark facilitates a nuanced evaluation of the trade-off between answer accuracy and privacy protection, thereby establishing a foundation for joint privacy-utility assessment in this critical domain.

Abstract

Large language models (LLMs) are increasingly applied to cybersecurity question answering (QA) for critical tasks such as incident response and vulnerability analysis. However, real-world operational contexts, including system logs and network configurations, inherently contain sensitive identifiers, e.g., IP addresses, host names, and user accounts. Processing this data with cloud-based models is often unsafe or infeasible in regulated environments. Furthermore, progress in privacy-preserving QA is hindered by the lack of annotated, context-rich datasets capable of jointly evaluating operational reasoning and privacy preservation. To address this gap, we introduce CYBERMASKQA, a privacy-aware QA benchmark covering key security domains. Unlike existing benchmarks that primarily test factual knowledge, CYBERMASKQA grounds questions in realistic organizational contexts with explicit causal dependencies among assets and privileges. Generated through a systematic pipeline, the dataset combines human-curated base scenarios with LLM-driven semantic expansion, annotating each instance with precise private entity labels to enable controlled information disclosure. Evaluations of QA accuracy and masking performance demonstrate the benchmark's utility for developing deployable, context-aware cybersecurity models and facilitating nuanced studies of privacy-utility trade-offs. Upon acceptance, we will release the dataset and the generation framework.

Problem

• privacy-aware
• cybersecurity question answering
• sensitive identifiers
• benchmark
• privacy preservation

Innovation

• privacy-aware QA
• cybersecurity benchmark
• controlled information disclosure
• context-rich dataset
• LLM-driven semantic expansion