ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

2026-01-30

AI Summary

This work addresses the limited generalization of existing GUI agents, which either rely on coordinate-based localization or suffer from data scarcity in coordinate-free approaches. The authors propose ToolTok, a framework that models GUI interaction as a multi-step tool-use sequence. By introducing learnable tool-token embeddings aligned with human interaction patterns and a semantic anchoring mechanism, ToolTok achieves strong generalization with minimal data. Integrated with a curriculum learning strategy—progressing from tool-definition question answering and textual tool selection to simplified visual path planning—and fine-tuning of a large language model, ToolTok outperforms same-scale (4B) baselines on multiple benchmarks using less than 1% of the training data, matching the performance of a 235B-parameter model and demonstrating exceptional generalization to unseen interfaces.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training&inference code is open-source at https://github.com/ZephinueCode/ToolTok.

Problem

• GUI agents
• generalization
• data scarcity
• input resolution
• visual grounding

Innovation

• tool tokenization
• multi-step pathfinding
• semantic anchuing
• curriculum learning
• GUI agents