FormGym: Doing Paperwork with Agents

2025-06-17

AI Summary

This paper addresses the challenge of automating form filling for pure-image forms—those lacking OCR, text layers, or DOM structure. To this end, we introduce FormGym, the first dedicated benchmark for vision-only form completion, comprising 55 documents, 432 fields, and three task categories that require multimodal understanding, key information retrieval, and tool invocation from visual inputs alone. Methodologically, we propose FieldFinder—a lightweight, vision-based field localization tool—and integrate it with a multimodal large model (VLA), a GUI agent, a vision-localization enhancement module, structured knowledge injection, and a tool-calling mechanism. Our contributions are: (1) the first fine-grained evaluation framework for pure-image form filling; and (2) FieldFinder, which boosts field localization accuracy from 2% to 56%, substantially outperforming baseline VLA (<1%) and GUI agents (10.6–68.0%).

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Completing paperwork is a challenging and time-consuming problem. Form filling is especially challenging in the pure-image domain without access to OCR, typeset PDF text, or a DOM. For computer agents, it requires multiple abilities, including multi-modal understanding, information retrieval, and tool-use. We present a novel form-filling benchmark consisting of 432 fields spread across 55 documents and 3 tasks, requiring knowledge of 236 features per user. We find that baseline VLAs achieve less than 1% accuracy in most cases, primarily due to poor localization ability. GUI agents also struggle, scoring between 10.6-68.0% despite high cost and latency. Therefore, we also contribute FieldFinder, a tool to assist LLMs in identifying where to place text on a form. With FieldFinder, all models achieve equal or better performance in all six study conditions, with a maximum increase from 2% to 56%.

Problem

• Challenges in completing paperwork without OCR or text access
• Low accuracy of baseline agents in form-filling tasks
• Need for improved localization tools in form-filling processes

Innovation

• Multi-modal understanding for form completion
• FieldFinder tool for text placement
• Benchmark with 432 fields across documents