🤖 AI Summary
This study addresses the challenge of efficiently analyzing large-scale, open-ended user feedback, which hinders effective user experience evaluation and product iteration. The authors propose a novel approach that integrates multi-label classification with generative artificial intelligence (GenAI) to automatically perform topic labeling, summarization, and sentiment analysis of user comments. This work represents the first effort to synergistically combine multi-label supervised learning with GenAI for user feedback analysis. Importantly, the findings reveal that sentiment analysis alone is insufficient as a proxy for explicit satisfaction measurement. By enabling scalable and fine-grained insights into user experience, the proposed method offers a robust technical pathway for practitioners and researchers seeking to derive actionable intelligence from unstructured feedback data.
📝 Abstract
In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.