LLM-Powered Analysis of IoT User Reviews: Tracking and Ranking Security and Privacy Concerns

2026-01-01

arXiv.org

AI Summary

This work addresses the challenge of effectively identifying and tracking security and privacy (S&P) concerns expressed by users in Internet of Things (IoT) product reviews—a critical yet underexplored area that hinders targeted system improvements. To this end, we propose an automated analysis framework leveraging large language models, which uniquely integrates dynamic few-shot prompting with contextual reasoning capabilities. Our approach employs a Classifier-Rationalizer-Categorizer pipeline coupled with a Thematic Mapper to achieve high-precision identification and thematic categorization of S&P-related comments. Evaluated on 91K Amazon IoT device reviews, the framework attains over 97% precision and recall in detecting the 5% of comments related to S&P—yielding a 15–70× increase in detected relevant content compared to prior methods—and uncovers long-overlooked issues in multi-user contexts, such as account isolation and data access control.

Technology Category

• Natural Language Processing: Safety and Robustness
• Application Domains: Internet of Things, Sensor Networks & Smart Cities
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Being able to understand the security and privacy (S&P) concerns of IoT users brings benefits to both developers and users. To learn about users'views, we examine Amazon IoT reviews - one of the biggest IoT markets. This work presents a state-of-the-art methodology to identify and categorize reviews in which users express S&P concerns. We developed an automated pipeline by fine-tuning GPT-3.5-Turbo to build two models: the Classifier-Rationalizer-Categorizer and the Thematic Mapper. By leveraging dynamic few-shot prompting and the model's large context size, our pipeline achieved over 97% precision and recall, significantly outperforming keyword-based and classical ML methods. We applied our pipeline to 91K Amazon reviews about fitness trackers, smart speakers and cameras, over multiple years. We found that on average 5% contained S&P concerns, while security camera exhibited the highest prevalence at 10%. Our method detected significantly more S&P-relevant reviews than prior works: 15x more for fitness trackers, 29% more for smart speakers, and 70% more for cameras. Our longitudinal analysis reveals that concerns like surveillance and data control have persisted for years, suggesting limited industry progress. We demonstrate that across all device types, users consistently demand more precise control over what data is collected and shared. We uncover challenges in multi-user and multi-device interactions, identifying two previously unreported themes concerning inadequate controls for account separation and data access. These findings, ranging from broad persistent trends to specific instances of customer loss, offer actionable insights for developers to improve user satisfaction and trust.

Problem

• IoT
• security and privacy
• user reviews
• LLM
• concern tracking

Innovation

• LLM-powered analysis
• dynamic few-shot prompting
• security and privacy concerns
• automated review classification
• IoT user feedback