Decoding User Concerns in AI Health Chatbots: An Exploration of Security and Privacy in App Reviews

2025-01-31

AI Summary

This study addresses users’ prominent concerns regarding data security and privacy (SPR) in AI-powered health chatbots, systematically analyzing app user reviews to identify critical trust gaps—including insufficient transparency, inadequate informed consent, and data misuse. Methodologically, it integrates fine-tuned BART, Gemini GenAI prompt engineering, and manual thematic coding, validated through rigorous inter-rater cross-validation to establish a hybrid evaluation framework. It provides the first empirical assessment of large language models’ applicability boundaries for SPR issue identification in health contexts: Gemini achieves classification accuracy comparable to human annotators, and over 68% of privacy-related reviews specifically cite transparency and consent deficiencies. The study contributes a practical, actionable SPR audit checklist and evidence-based communication strategies for developers, thereby advancing methodological rigor and pragmatic guidance for trustworthy AI governance in digital health.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Privacy & Security
• Humans and AI: AI for Accessibility

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

AI powered health chatbot applications are increasingly utilized for personalized healthcare services, yet they pose significant challenges related to user data security and privacy. This study evaluates the effectiveness of automated methods, specifically BART and Gemini GenAI, in identifying security privacy related (SPR) concerns within these applications' user reviews, benchmarking their performance against manual qualitative analysis. Our results indicate that while Gemini's performance in SPR classification is comparable to manual labeling, both automated methods have limitations, including the misclassification of unrelated issues. Qualitative analysis revealed critical user concerns, such as data collection practices, data misuse, and insufficient transparency and consent mechanisms. This research enhances the understanding of the relationship between user trust, privacy, and emerging mobile AI health chatbot technologies, offering actionable insights for improving security and privacy practices in AI driven health chatbots. Although exploratory, our findings highlight the necessity for rigorous audits and transparent communication strategies, providing valuable guidance for app developers and vendors in addressing user security and privacy concerns.

Problem

• AI health chatbots
• privacy concerns
• automated tool limitations

Innovation

• Gemini GenAI
• Privacy Concerns
• AI Chatbot Analysis