Crepe: A Mobile Screen Data Collector Using Graph Query

2024-06-23

arXiv.org

AI Summary

Academic researchers face significant challenges in collecting mobile screen data—including limited access due to proprietary platform restrictions, stringent commercial monopolies, and heightened privacy compliance requirements. Existing open-source frameworks predominantly focus on sensor data and lack robust, privacy-compliant, and flexible mechanisms for capturing screen content. Method: We propose Crepe, the first no-code Android screen data collection tool designed specifically for academic research. It introduces a novel graph-query-based UI structural representation to enable semantic identification and high-precision localization of screen elements. Crepe integrates declarative demonstration learning, on-device processing, and a permission sandbox to ensure informed consent and real-time user opt-out. Contribution/Results: Empirical evaluation across diverse applications demonstrates that Crepe achieves zero-configuration extraction of dynamic text and UI controls with high accuracy, effectively circumventing data monopolies while enabling privacy-preserving screen-content research.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Collecting mobile datasets remains challenging for academic researchers due to limited data access and technical barriers. Commercial organizations often possess exclusive access to mobile data, leading to a"data monopoly"that restricts the independence of academic research. Existing open-source mobile data collection frameworks primarily focus on mobile sensing data rather than screen content, which is crucial for various research studies. We present Crepe, a no-code Android app that enables researchers to collect information displayed on screen through simple demonstrations of target data. Crepe utilizes a novel Graph Query technique which augments the structures of mobile UI screens to support flexible identification, location, and collection of specific data pieces. The tool emphasizes participants' privacy and agency by providing full transparency over collected data and allowing easy opt-out. We designed and built Crepe for research purposes only and in scenarios where researchers obtain explicit consent from participants. Code for Crepe will be open-sourced to support future academic research data collection.

Problem

• Overcoming mobile data access barriers for academic research
• Enabling screen content collection via no-code Android app
• Addressing data monopoly and privacy concerns in research

Innovation

• No-code Android app for screen data collection
• Graph Query technique for flexible UI data identification
• Ensures participant privacy with transparency and opt-out