RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

2026-02-28

AI Summary

Existing evaluation frameworks struggle to effectively assess the true capabilities of large language models in complex text synthesis tasks such as outlining, drafting, and editing. To address this limitation, this work proposes an agent-based evaluation paradigm that decouples core competencies in text synthesis through reverse engineering. The authors introduce a task suite comprising four stages—Cloze, Edit, Expand, and End-to-End—corresponding to outlining, drafting, reviewing, and refinement, and release C3EBench, a benchmark containing 1,258 human expert writing samples. Experimental results across 14 prominent large language models demonstrate that reasoning ability is a stronger determinant of text synthesis performance than raw generative capacity, and that a powerful reasoner can substantially enhance the output quality of a weaker generator.

Technology Category

• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval.

Problem

• LLM evaluation
• text synthesis
• reasoning agents
• synthesis operations
• benchmarking

Innovation

• Reasoning Agents
• Text Synthesis Evaluation
• Agentic Framework
• C3EBench
• LLM Capability Decomposition