OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas

2025-11-23

AI Summary

Current large language models (LLMs) exhibit unclear generalization capabilities for arbitrary-pattern structured output generation—e.g., information extraction, table generation, and function calling. To address this, we propose OmniStruct: the first comprehensive, multi-task, multi-schema text-to-structure evaluation benchmark. We further design a synthetic task-based data generation methodology to produce high-quality, schema-agnostic training data. Building upon this, we introduce a unified text-to-structure modeling paradigm enabling efficient fine-tuning of small open-source models. Experiments demonstrate that a Qwen2-1.5B model fine-tuned solely on synthetic data outperforms GPT-4o across all OmniStruct tasks—validating the feasibility of lightweight models for general-purpose structured generation. Our core contributions are threefold: (1) the OmniStruct benchmark, (2) a scalable, schema-agnostic synthetic data generation framework, and (3) an efficient fine-tuning paradigm for compact models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The ability of Large Language Models (LLMs) to generate structured outputs that follow arbitrary schemas is crucial to a wide range of downstream tasks that require diverse structured representations of results such as information extraction, table generation, and function calling. While modern LLMs excel in generating unstructured responses in natural language, whether this advancement translates to a strong performance on text-to-structure tasks remains unclear. To bridge this gap, we first introduce OmniStruct, a comprehensive benchmark for assessing LLMs' capabilities on diverse text-to-structure tasks such as information extraction, table generation, and function calling. We build OmniStruct by identifying existing datasets across a wide range of tasks that are suitable for a structured answer format, and adapting them under a unified text-to-structure problem setting. To facilitate the development of efficient text-to-structure models, we collect high-quality training data via synthetic task generation. Without using any supervised data for OmniStruct tasks, our experiments demonstrate the possibility of fine-tuning much smaller models on synthetic data into universal structured generation models that can rival the performance of GPT-4o.

Problem

• Assessing LLM capabilities for diverse text-to-structure generation tasks
• Developing universal structured generation models using synthetic training data
• Enabling structured outputs for information extraction and table generation

Innovation

• Comprehensive benchmark for diverse text-to-structure tasks
• Synthetic task generation for high-quality training data
• Fine-tuning smaller models to rival GPT-4o performance