DocReward: A Document Reward Model for Structuring and Stylizing

2025-10-13

AI Summary

Existing automated documentation generation methods overemphasize textual quality while neglecting visual structure and stylistic consistency, resulting in suboptimal readability and professionalism, and lack domain-specific reward models for structural and stylistic evaluation. Method: We propose DocReward—the first reward model explicitly designed to assess the structural integrity and stylistic professionalism of technical documentation. To train it, we construct DocPair, a large-scale, multi-domain dataset comprising 117,000 document pairs annotated for human-preferred structural and stylistic variations under content-equivalence constraints. DocReward is trained using the Bradley–Terry loss function on human pairwise preference rankings. Results: Experiments show DocReward achieves 30.6% and 19.4% higher accuracy than GPT-4o and GPT-5, respectively, on professional documentation ranking tasks. In out-of-distribution generative preference judgment, DocReward attains a 60.8% win rate—significantly outperforming GPT-5 (37.7%).

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Structured Learning
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent advances in agentic workflows have enabled the automation of tasks such as professional document generation. However, they primarily focus on textual quality, neglecting visual structure and style, which are crucial for readability and engagement. This gap arises mainly from the absence of suitable reward models to guide agentic workflows toward producing documents with stronger structural and stylistic quality. To address this, we propose DocReward, a document reward model that evaluates documents based on their structure and style. We construct a multi-domain dataset DocPair of 117K paired documents, covering 32 domains and 267 document types, each including a high- and low-professionalism document with identical content but different structure and style. This enables the model to evaluate professionalism comprehensively, and in a textual-quality-agnostic way. DocReward is trained using the Bradley-Terry loss to score documents, penalizing predictions that contradict the annotated ranking. To assess the performance of reward models, we create a test dataset containing document bundles ranked by well-educated human evaluators. Notably, DocReward outperforms GPT-4o and GPT-5 in accuracy by 30.6 and 19.4 percentage points, respectively, demonstrating its superiority over baselines. In an extrinsic evaluation of document generation, DocReward achieves a significantly higher win rate of 60.8%, compared to GPT-5's 37.7% win rate, demonstrating its utility in guiding generation agents toward producing human-preferred documents.

Problem

• Addresses the lack of visual structure and style in automated document generation
• Proposes a reward model to evaluate document professionalism beyond text quality
• Improves document generation by guiding agents toward human-preferred layouts

Innovation

• DocReward model evaluates document structure and style
• Trained on multi-domain dataset using Bradley-Terry loss
• Outperforms GPT models in document professionalism assessment