Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry

2025-01-14

AI Summary

Navigation apps frequently receive repetitive user reviews, and manual response generation is inefficient. Method: This paper proposes a semi-automated explanation generation system that (1) constructs a hierarchical taxonomy to identify and prioritize user explanation needs, and (2) designs a rule-based team assignment algorithm that integrates multi-source knowledge—including internal historical responses and external support documentation—to enable cross-platform (Google Play & App Store) collaborative response generation. Contribution/Results: To the best of our knowledge, this is the first industrial deployment integrating multi-source external knowledge retrieval deeply into both review classification and response generation pipelines. Experiments show a team assignment accuracy of 79.2% and a significant reduction in manual response time. The system was validated with a real four-member support team, demonstrating its feasibility and practical value in production environments.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Multiagent Systems: Teamwork
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Providing explanations in response to user reviews is a time-consuming and repetitive task for companies, as many reviews present similar issues requiring nearly identical responses. To improve efficiency, this paper proposes a semi-automated approach to managing explanation needs in user reviews. The approach leverages taxonomy categories to classify reviews and assign them to relevant internal teams or sources for responses. 2,366 app reviews from the Google Play Store and Apple App Store were scraped and analyzed using a word and phrase filtering system to detect explanation needs. The detected needs were categorized and assigned to specific internal teams at the company Graphmasters GmbH, using a hierarchical assignment strategy that prioritizes the most relevant teams. Additionally, external sources, such as existing support articles and past review responses, were integrated to provide comprehensive explanations. The system was evaluated through interviews and surveys with the Graphmasters support team, which consists of four employees. The results showed that the hierarchical assignment method improved the accuracy of team assignments, with correct teams being identified in 79.2% of cases. However, challenges in interrater agreement and the need for new responses in certain cases, particularly for Apple App Store reviews, were noted. Future work will focus on refining the taxonomy and enhancing the automation process to reduce manual intervention further.

Problem

• Navigation Software
• User Feedback Management
• Automation

Innovation

• Semi-automatic Processing
• Classification Rules
• Team Allocation Optimization