GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph

2026-01-24

Journal of Intelligent Computing and Networking

AI Summary

This work proposes a novel paradigm for mobile GUI agents that integrates application-specific knowledge graphs with one-step LLM reasoning to overcome the high latency and inefficiency of existing approaches relying on multi-round query-execution loops. During an offline phase, a knowledge graph encoding page functionalities and navigation rules is constructed; at runtime, this graph guides the LLM to generate a complete action sequence in a single inference step. Execution reliability is ensured through dynamic interface feedback and a validation mechanism. By uniquely combining knowledge graphs with one-step LLM reasoning, the method significantly improves task completion rates while substantially reducing both latency and the number of LLM invocations, outperforming state-of-the-art baselines such as Mind2Web and AutoDroid on the DroidTask benchmark.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Multiagent Systems: Modeling other Agents

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Mobile graphical user interface (GUI) agents are designed to automate everyday tasks on smartphones. Recent advances in large language models (LLMs) have significantly enhanced the capabilities of mobile GUI agents. However, most LLM-powered mobile GUI agents operate in stepwise query-act loops, which incur high latency due to repeated LLM queries. We present GraphPilot, a mobile GUI agent that leverages knowledge graphs of the target apps to complete user tasks in almost one LLM query. GraphPilot operates in two complementary phases to enable efficient and reliable LLM-powered GUI task automation. In the offline phase, it explores target apps, records and analyzes interaction history, and constructs an app-specific knowledge graph that encodes functions of pages and elements as well as transition rules for each app. In the online phase, given an app and a user task, it leverages the knowledge graph of the given app to guide the reasoning process of LLM. When the reasoning process encounters uncertainty, GraphPilot dynamically requests the HTML representation of the current interface to refine subsequent reasoning. Finally, a validator checks the generated sequence of actions against the transition rules in the knowledge graph, performing iterative corrections to ensure it is valid. The structured, informative information in the knowledge graph allows the LLM to plan the complete sequence of actions required to complete the user task. On the DroidTask benchmark, GraphPilot improves task completion rate over Mind2Web and AutoDroid, while substantially reducing latency and the number of LLM queries.

Problem

• GUI task automation
• large language models
• latency
• mobile agents
• stepwise reasoning

Innovation

• knowledge graph
• one-step LLM reasoning
• GUI task automation
• mobile agent
• structured planning