From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents

2025-12-15

AI Summary

Large language model (LLM) agents incur excessive token consumption and struggle to balance semantic completeness with efficiency in UI navigation due to suboptimal UI representations. Method: This paper proposes the first automated program synthesis framework dedicated to UI representation optimization. It introduces a UI-specific domain-specific language (DSL) to constrain the program search space; designs an LLM-based iterative refinement mechanism jointly optimizing for correctness and efficiency via dual reward modeling; and adopts a lightweight, plug-and-play architecture enabling zero-modification integration with existing LLM agents. Results: Evaluated on three major UI navigation benchmarks (Android and Web), the framework achieves 48.7%–55.8% token compression across five mainstream LLMs, with negligible runtime overhead and maintained or improved task performance. The approach has been successfully deployed in industrial-scale UI automation within WeChat.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Learning to Search
• Humans and AI: Intelligent User Interfaces

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While Large Language Model (LLM) agents show great potential for automated UI navigation such as automated UI testing and AI assistants, their efficiency has been largely overlooked. Our motivating study reveals that inefficient UI representation creates a critical performance bottleneck. However, UI representation optimization, formulated as the task of automatically generating programs that transform UI representations, faces two unique challenges. First, the lack of Boolean oracles, which traditional program synthesis uses to decisively validate semantic correctness, poses a fundamental challenge to co-optimization of token efficiency and completeness. Second, the need to process large, complex UI trees as input while generating long, compositional transformation programs, making the search space vast and error-prone. Toward addressing the preceding limitations, we present UIFormer, the first automated optimization framework that synthesizes UI transformation programs by conducting constraint-based optimization with structured decomposition of the complex synthesis task. First, UIFormer restricts the program space using a domain-specific language (DSL) that captures UI-specific operations. Second, UIFormer conducts LLM-based iterative refinement with correctness and efficiency rewards, providing guidance for achieving the efficiency-completeness co-optimization. UIFormer operates as a lightweight plugin that applies transformation programs for seamless integration with existing LLM agents, requiring minimal modifications to their core logic. Evaluations across three UI navigation benchmarks spanning Android and Web platforms with five LLMs demonstrate that UIFormer achieves 48.7% to 55.8% token reduction with minimal runtime overhead while maintaining or improving agent performance. Real-world industry deployment at WeChat further validates the practical impact of UIFormer.

Problem

• Optimizes UI representations to reduce token usage for LLM agents
• Addresses lack of Boolean oracles for verifying UI transformation correctness
• Solves search space explosion when generating complex UI transformation programs

Innovation

• Automated UI transformation program synthesis framework
• Constraint-based optimization with structured task decomposition
• LLM-based iterative refinement with efficiency rewards