From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

2026-04-30

AI Summary

This work addresses the challenge that current large language models struggle to precisely follow fine-grained instructions in text generation tasks, compounded by the inadequacy of existing evaluation benchmarks, which are overly coarse. To overcome this limitation, the authors propose WEval, a fine-grained evaluation pipeline, and WRL, a reinforcement learning training framework. By constructing evaluation data spanning multiple categories and requirement types, and selectively curating positive and negative samples that either satisfy or violate specific instruction constraints, the approach enables accurate modeling and training of writing-oriented reward models. This study introduces, for the first time, a fine-grained reward mechanism tailored to concrete writing requirements, achieving significant performance gains across multiple writing benchmarks and demonstrating strong generalization capabilities.

Abstract

Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.

Problem

• reward modeling
• writing-centric generation
• fine-grained evaluation
• requirement adherence
• language models

Innovation

• fine-grained reward modeling
• writing-centric generation
• WEval
• WRL
• instruction adherence