OrionBench: A Benchmark for Chart and Human-Recognizable Object Detection in Infographics

2025-05-23

AI Summary

Current vision-language models (VLMs) exhibit insufficient fine-grained localization capability for chart structures and human-recognizable objects (HROs) in infographics, severely limiting infographic understanding performance. To address this, we introduce InfographicDet—the first fine-grained detection benchmark explicitly designed for charts and HROs in infographics—and propose “Thinking-with-Boxes,” a novel reasoning paradigm that deeply integrates high-precision object detection into the infographic comprehension pipeline. Our methodology innovatively combines model-in-the-loop annotation, programmatically synthesized data generation, bounding-box-supervised learning, and spatial-semantic alignment prompt enhancement. The benchmark comprises 105,000 infographics and over 6.9 million high-quality bounding box annotations. Experiments demonstrate substantial improvements in VLM accuracy on chart question answering, with successful cross-task transfer to document layout analysis and UI element detection.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Given the central role of charts in scientific, business, and communication contexts, enhancing the chart understanding capabilities of vision-language models (VLMs) has become increasingly critical. A key limitation of existing VLMs lies in their inaccurate visual grounding of infographic elements, including charts and human-recognizable objects (HROs) such as icons and images. However, chart understanding often requires identifying relevant elements and reasoning over them. To address this limitation, we introduce OrionBench, a benchmark designed to support the development of accurate object detection models for charts and HROs in infographics. It contains 26,250 real and 78,750 synthetic infographics, with over 6.9 million bounding box annotations. These annotations are created by combining the model-in-the-loop and programmatic methods. We demonstrate the usefulness of OrionBench through three applications: 1) constructing a Thinking-with-Boxes scheme to boost the chart understanding performance of VLMs, 2) comparing existing object detection models, and 3) applying the developed detection model to document layout and UI element detection.

Problem

• Improving chart and object detection in infographics for VLMs
• Addressing inaccurate visual grounding of infographic elements
• Creating a benchmark for accurate object detection models

Innovation

• Combines model-in-the-loop and programmatic annotation methods
• Introduces OrionBench with 105k infographics and 6.9M annotations
• Enables Thinking-with-Boxes for VLM chart understanding enhancement