Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic
This study investigates topic-level patterns between user prompts and LLM responses in the LMSYS-Chat-1M dataset and their correlation with human model preferences. Method: We pioneer the application of BERTopic to multilingual LLM comparative evaluation data, integrating dialogue cleaning, multilingual preprocessing, and topic distribution visualization to construct a model–topic preference matrix. Contribution/Results: We identify 29 semantically coherent topics and discover consistent user preference advantages for specific LLMs across domains such as technology, programming, and ethics—revealing a topic-dependent distribution of model strengths. This work establishes an interpretable, topic-level analytical framework for LLM capability assessment and enables domain-aware model selection and targeted fine-tuning, thereby advancing personalized LLM deployment.