Testing Practices, Challenges, and Developer Perspectives in Open-Source IoT Platforms

2025-02-11

AI Summary

IoT platforms exhibit high heterogeneity, diverse communication protocols, and complex cloud–edge–device collaboration, leading to severely lagging testing practices. Method: This paper presents the first systematic empirical study of testing practices in open-source smart home platforms (OpenHAB and Home Assistant), adopting a dual-dimension framework integrating code-based evidence (via static test-code analysis) and developer cognition (via structured surveys and qualitative interviews with 80 active developers). Contribution/Results: We uncover stark disparities in test coverage (0.04 for OpenHAB vs. 0.42 for Home Assistant), widespread reliance on unit testing, and consistent neglect of manual and end-to-end testing. Furthermore, we identify and characterize ten IoT-specific testing challenges. This work provides the first large-scale empirical foundation to inform the design of IoT testing toolchains, standardization efforts, and engineering education.

Technology Category

• Application Domains: Internet of Things, Sensor Networks & Smart Cities
• Intelligent Robots: Other Foundations and Applications
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

As the popularity of Internet of Things (IoT) platforms grows, users gain unprecedented control over their homes, health monitoring, and daily task automation. However, the testing of software for these platforms poses significant challenges due to their diverse composition, e.g., common smart home platforms are often composed of varied types of devices that use a diverse array of communication protocols, connections to mobile apps, cloud services, as well as integration among various platforms. This paper is the first to uncover both the practices and perceptions behind testing in IoT platforms, particularly open-source smart home platforms. Our study is composed of two key components. First, we mine and empirically analyze the code and integrations of two highly popular and well-maintained open-source IoT platforms, OpenHab and HomeAssitant. Our analysis involves the identification of functional and related test methods based on the focal method approach. We find that OpenHab has only 0.04 test ratio ($approx 4K$ focal test methods from $approx 76K$ functional methods) in Java files, while HomeAssitant exhibits higher test ratio of $0.42$, which reveals a significant dearth of testing. Second, to understand the developers' perspective on testing in IoT, and to explain our empirical observations, we survey 80 open-source developers actively engaged in IoT platform development. Our analysis of survey responses reveals a significant focus on automated (unit) testing, and a lack of manual testing, which supports our empirical observations, as well as testing challenges specific to IoT. Together, our empirical analysis and survey yield 10 key findings that uncover the current state of testing in IoT platforms, and reveal key perceptions and challenges. These findings provide valuable guidance to the research community in navigating the complexities of effectively testing IoT platforms.

Problem

• Explores testing practices in open-source IoT platforms
• Identifies challenges in IoT software testing diversity
• Surveys developer perspectives on IoT testing methodologies

Innovation

• Analyzes OpenHab and HomeAssitant testing
• Surveys 80 IoT developers
• Focuses on automated unit testing