ProfiLLM: An LLM-Based Framework for Implicit Profiling of Chatbot Users

2025-06-16

AI Summary

Large language model (LLM)-based chatbots struggle to dynamically adapt to users’ implicit characteristics—such as technical proficiency and learning style—in specialized domains like IT and cybersecurity. Method: We propose the first implicit, dynamic user profiling framework tailored for IT security (ITSec). It comprises: (i) a structured, cross-domain professional competency taxonomy; (ii) lightweight, single-turn implicit user modeling; and (iii) an LLM-driven persona simulation and synthetic data generation paradigm integrated with online profile updating. Contribution/Results: Evaluated on 1,760 synthetic dialogues, our approach reduces per-turn technical competency prediction error by 55–65%; subsequent interactions further refine accuracy through continuous adaptation. The framework is fully open-sourced, including implementation code, the ITSec-specific taxonomy, and the dialogue dataset.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

Despite significant advancements in conversational AI, large language model (LLM)-powered chatbots often struggle with personalizing their responses according to individual user characteristics, such as technical expertise, learning style, and communication preferences. This lack of personalization is particularly problematic in specialized knowledge-intense domains like IT/cybersecurity (ITSec), where user knowledge levels vary widely. Existing approaches for chatbot personalization primarily rely on static user categories or explicit self-reported information, limiting their adaptability to an evolving perception of the user's proficiency, obtained in the course of ongoing interactions. In this paper, we propose ProfiLLM, a novel framework for implicit and dynamic user profiling through chatbot interactions. This framework consists of a taxonomy that can be adapted for use in diverse domains and an LLM-based method for user profiling in terms of the taxonomy. To demonstrate ProfiLLM's effectiveness, we apply it in the ITSec domain where troubleshooting interactions are used to infer chatbot users' technical proficiency. Specifically, we developed ProfiLLM[ITSec], an ITSec-adapted variant of ProfiLLM, and evaluated its performance on 1,760 human-like chatbot conversations from 263 synthetic users. Results show that ProfiLLM[ITSec] rapidly and accurately infers ITSec profiles, reducing the gap between actual and predicted scores by up to 55--65% after a single prompt, followed by minor fluctuations and further refinement. In addition to evaluating our new implicit and dynamic profiling framework, we also propose an LLM-based persona simulation methodology, a structured taxonomy for ITSec proficiency, our codebase, and a dataset of chatbot interactions to support future research.

Problem

• Personalizing chatbot responses based on user characteristics
• Adapting to varying user knowledge in IT/cybersecurity domains
• Improving dynamic user profiling without explicit self-reports

Innovation

• LLM-based implicit user profiling framework
• Dynamic adaptation to user proficiency changes
• Taxonomy-driven profiling for diverse domains