LLMAID: Identifying AI Capabilities in Android Apps with LLMs

2025-11-24

AI Summary

Existing Android AI capability identification relies on manual rules, suffering from high costs, poor adaptability, and limited coverage of emerging AI technologies. To address this, we propose LLMAID—the first systematic, end-to-end, large language model (LLM)-integrated framework for AI capability identification, comprising four stages: knowledge base interaction, candidate component extraction, fine-grained AI functionality analysis, and service summary generation. Evaluated on 4,201 real-world Android applications, LLMAID increases AI app detection by 242% over conventional methods, achieving precision and recall both exceeding 90%. We find that 54.80% of identified AI functionalities pertain to computer vision. Moreover, developer evaluations confirm that LLMAID-generated service summaries attain significantly higher usability ratings. This work establishes a novel paradigm for sensing, governing, and studying the evolution of mobile AI ecosystems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility
• Computer Vision: Large Vision Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in artificial intelligence (AI) and its widespread integration into mobile software applications have received significant attention, highlighting the growing prominence of AI capabilities in modern software systems. However, the inherent hallucination and reliability issues of AI continue to raise persistent concerns. Consequently, application users and regulators increasingly ask critical questions such as: Does the application incorporate AI capabilities? and What specific types of AI functionalities are embedded? Preliminary efforts have been made to identify AI capabilities in mobile software; however, existing approaches mainly rely on manual inspection and rule-based heuristics. These methods are not only costly and time-consuming but also struggle to adapt advanced AI techniques. To address the limitations of existing methods, we propose LLMAID (Large Language Model for AI Discovery). LLMAID includes four main tasks: (1) candidate extraction, (2) knowledge base interaction, (3) AI capability analysis and detection, and (4) AI service summarization. We apply LLMAID to a dataset of 4,201 Android applications and demonstrate that it identifies 242% more real-world AI apps than state-of-the-art rule-based approaches. Our experiments show that LLM4AID achieves high precision and recall, both exceeding 90%, in detecting AI-related components. Additionally, a user study indicates that developers find the AI service summaries generated by LLMAID to be more informative and preferable to the original app descriptions. Finally, we leverage LLMAID to perform an empirical analysis of AI capabilities across Android apps. The results reveal a strong concentration of AI functionality in computer vision (54.80%), with object detection emerging as the most common task (25.19%).

Problem

• Identifying AI capabilities embedded within Android mobile applications
• Overcoming limitations of manual inspection and rule-based detection methods
• Analyzing specific types and distributions of AI functionalities in apps

Innovation

• Uses large language models for AI capability identification
• Automates analysis through knowledge base interaction
• Generates informative AI service summaries for developers