DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysis

2026-01-29

AI Summary

Existing data analysis agents struggle to effectively integrate structured data with “zombie data” embedded in unstructured visual documents—such as scanned reports and invoices—leading to fragmented cross-modal information. To address this, this work proposes DataCrossBench, a benchmark, and DataCrossAgent, a novel framework that leverages a human-in-the-loop reverse synthesis pipeline to construct 200 end-to-end cross-modal tasks. The framework employs a divide-and-conquer multi-agent collaboration mechanism, integrating three core workflows: Intra-source Deep Exploration, Key Source Identification, and Contextual Cross-pollination, augmented by an innovative reReAct mechanism for verifiable code generation and debugging. Experiments demonstrate that DataCrossAgent improves factual accuracy by 29.7% over GPT-4o and exhibits significantly enhanced robustness on challenging tasks, effectively unlocking fragmented visual data for deep cross-modal analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

In real-world data science and enterprise decision-making, critical information is often fragmented across directly queryable structured sources (e.g., SQL, CSV) and"zombie data"locked in unstructured visual documents (e.g., scanned reports, invoice images). Existing data analytics agents are predominantly limited to processing structured data, failing to activate and correlate this high-value visual information, thus creating a significant gap with industrial needs. To bridge this gap, we introduce DataCross, a novel benchmark and collaborative agent framework for unified, insight-driven analysis across heterogeneous data modalities. DataCrossBench comprises 200 end-to-end analysis tasks across finance, healthcare, and other domains. It is constructed via a human-in-the-loop reverse-synthesis pipeline, ensuring realistic complexity, cross-source dependency, and verifiable ground truth. The benchmark categorizes tasks into three difficulty tiers to evaluate agents'capabilities in visual table extraction, cross-modal alignment, and multi-step joint reasoning. We also propose the DataCrossAgent framework, inspired by the"divide-and-conquer"workflow of human analysts. It employs specialized sub-agents, each an expert on a specific data source, which are coordinated via a structured workflow of Intra-source Deep Exploration, Key Source Identification, and Contextual Cross-pollination. A novel reReAct mechanism enables robust code generation and debugging for factual verification. Experimental results show that DataCrossAgent achieves a 29.7% improvement in factuality over GPT-4o and exhibits superior robustness on high-difficulty tasks, effectively activating fragmented"zombie data"for insightful, cross-modal analysis.

Problem

• cross-modal data analysis
• heterogeneous data
• unstructured visual data
• zombie data
• data analytics agents

Innovation

• cross-modal data analysis
• heterogeneous data integration
• agent framework
• visual document understanding
• reReAct mechanism