Problem
Research questions and friction points this paper is trying to address.
vision-language models
data annotation
structured visual reasoning
annotation-tool bottleneck
grounded representation
From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models
🤖 AI Summary
Current visual language model (VLM) training is hindered by the scarcity of high-quality annotated data that jointly captures unified spatial coordinates, open-vocabulary semantics, structural attributes, and topological relationships. Moreover, conventional annotation tools suffer from limited expressiveness, a disconnect between annotation and training pipelines, and poor reusability. To address these challenges, this work proposes ScreenAnnotator, which introduces a unified atomic annotation schema and integrates an online policy-based annotation loop with an embedded Bayesian verifier alongside a template-driven multitask data synthesis mechanism. This framework enables efficient and reusable construction of visual reasoning datasets. Evaluated on flowchart and GUI screenshot annotations, the system achieves acceptance rates of 99.7% and 77%, respectively, with consistently decreasing per-image annotation time. Fine-tuning VLMs on the generated data yields a 76.1% accuracy on flowchart understanding tasks, representing an absolute improvement of 35.1 percentage points.
📝 Abstract
Vision-language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual reasoning. Training models for such advanced capabilities demands a new genre of data that seamlessly unifies spatial coordinates, open-vocabulary descriptions, structured attributes, and topological relationships into a singular representation. However, existing data annotation tools fundamentally fail to meet these intricate demands, suffering from three systematic bottlenecks: limited expressiveness, severe annotation-training decoupling, and poor data reusability. To bridge this infrastructure gap, we introduce an open-source annotation tool, ScreenAnnotator. First, we define a unified annotation atom schema that binds spatial, semantic, and structural primitives into a single unit. Second, we implement an on-policy annotation loop embedded with a Bayesian Annotation Verifier (BAV). Finally, we design a template-driven multi-task data synthesis process dynamically transforms static atoms into diverse multi-dimensional reasoning tasks, eliminating redundant re-annotation. The on-policy loop drives the annotation accept rate to nearly 100% on flowcharts and 77% on GUI screenshots, while steadily reducing per-image annotation time as labeled data accumulate. In the flowchart scenario, fine-tuning a VLM yields 76.1% average accuracy, which is a 35.1% point absolute gain. Our code is available at: https://github.com/WnQinm/Annotator.
Innovation
Methods, ideas, or system contributions that make the work stand out.
on-policy annotation
vision-language models
structured visual reasoning
Bayesian Annotation Verifier
annotation atom
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
