Evaluating the Impact of Personalization in Conversational Cybersecurity Assistants

2026-09-15

AI Summary

研究通过四种个性化策略提高基于大语言模型的网络安全助手的回答效果，特别是对话历史个性化策略在帮助性和遵循安全建议可能性上表现最佳。

Abstract

Users increasingly turn to Large Language Models to answer a variety of questions, including cybersecurity questions. We study how personalization strategies can help improve the effectiveness of answers to questions asked to an LLM-based cybersecurity assistant. Beyond accuracy, we focus on the understandability, actionability and, most importantly, motivating power of answers, given how often users fail to follow cybersecurity recommendations. Specifically, we investigate four personalization strategies, ranging from static user profiles to interaction-history-based personalization, using a corpus of 1,045 real-world cybersecurity questions and a 7-day deployment involving 57 participants and 1,066 user questions. Across both a large-scale automated LLM-based evaluation and human evaluation, conversation-based personalization is consistently favored in comparative ratings of perceived helpfulness and likelihood of following security advice. Importantly, the relative trends observed in the LLM-based evaluation align with those obtained from human evaluation, suggesting that LLM-based evaluation can provide a scalable mechanism for comparing personalization strategies before costly user studies. These results indicate that behavior-driven personalization is a promising direction for LLM-powered cybersecurity assistants and highlight the value of combining LLM-based and human evaluation when studying personalized language-model systems.

Problem

• Personalization
• Cybersecurity
• Large Language Models
• User Engagement
• Security Advice

Innovation

• personalization strategies
• conversation-based personalization
• LLM-based evaluation
• cybersecurity assistant