LLM-Cure: LLM-based Competitor User Review Analysis for Feature Enhancement

2024-09-24

arXiv.org

AI Summary

Mobile app continuous optimization faces challenges including inefficient user review analysis, lack of competitive benchmarking, and insufficient actionable improvement suggestions. To address these, we propose the first cross-app review understanding framework integrating competitive app comparison and large language models (LLMs). Our method employs LLM-driven fine-grained feature identification, semantic retrieval, and cross-app review alignment to automatically localize defects in the target app and retrieve high-rated reviews from competing apps to generate targeted functional improvement recommendations—establishing an end-to-end closed loop from problem detection to solution generation. Evaluated on 70 Android apps and over one million reviews, our approach achieves a 13% improvement in feature attribution F1-score and a 73% developer adoption rate for generated suggestions, significantly outperforming existing single-app analysis methods.

Technology Category

• Search and Optimization: Applications
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Feature Construction/Reformulation

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The exponential growth of the mobile app market underscores the importance of constant innovation and rapid response to user demands. As user satisfaction is paramount to the success of a mobile application (app), developers typically rely on user reviews, which represent user feedback that includes ratings and comments to identify areas for improvement. However, the sheer volume of user reviews poses challenges in manual analysis, necessitating automated approaches. Existing automated approaches either analyze only the target apps reviews, neglecting the comparison of similar features to competitors or fail to provide suggestions for feature enhancement. To address these gaps, we propose a Large Language Model (LLM)-based Competitive User Review Analysis for Feature Enhancement) (LLM-Cure), an approach powered by LLMs to automatically generate suggestion s for mobile app feature improvements. More specifically, LLM-Cure identifies and categorizes features within reviews by applying LLMs. When provided with a complaint in a user review, LLM-Cure curates highly rated (4 and 5 stars) reviews in competing apps related to the complaint and proposes potential improvements tailored to the target application. We evaluate LLM-Cure on 1,056,739 reviews of 70 popular Android apps. Our evaluation demonstrates that LLM-Cure significantly outperforms the state-of-the-art approaches in assigning features to reviews by up to 13% in F1-score, up to 16% in recall and up to 11% in precision. Additionally, LLM-Cure demonstrates its capability to provide suggestions for resolving user complaints. We verify the suggestions using the release notes that reflect the changes of features in the target mobile app. LLM-Cure achieves a promising average of 73% of the implementation of the provided suggestions.

Problem

• Automates analysis of competitor app reviews for feature enhancement
• Uses LLMs to categorize features and suggest improvements
• Addresses gaps in existing automated review analysis methods

Innovation

• LLM-based analysis of competitor user reviews
• Automated feature enhancement suggestions generation
• High accuracy in feature categorization and suggestions