Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification

2025-02-12

AI Summary

This work addresses the lack of systematic evaluation of large language models (LLMs) on multi-granularity text simplification—spanning lexical, syntactic, sentence-, and document-level tasks. We introduce the first unified benchmark covering all four granularity levels. Using lightweight and mainstream open- and closed-source LLMs, we comparatively evaluate them against traditional non-LLM simplification methods, employing both automatic metrics (BLEU, SARI) and multidimensional human assessment (readability, conciseness, meaning preservation). Results show that LLMs significantly outperform conventional approaches across all four granularities; notably, outputs from several models surpass human reference texts in quality, challenging the authority of existing “gold-standard” references. This study fills a critical gap in cross-granularity LLM-based simplification evaluation and establishes a comprehensive assessment framework that integrates automated metrics with human judgment—providing a new benchmark and methodological foundation for text simplification research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evaluation and Analysis

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Text simplification (TS) refers to the process of reducing the complexity of a text while retaining its original meaning and key information. Existing work only shows that large language models (LLMs) have outperformed supervised non-LLM-based methods on sentence simplification. This study offers the first comprehensive analysis of LLM performance across four TS tasks: lexical, syntactic, sentence, and document simplification. We compare lightweight, closed-source and open-source LLMs against traditional non-LLM methods using automatic metrics and human evaluations. Our experiments reveal that LLMs not only outperform non-LLM approaches in all four tasks but also often generate outputs that exceed the quality of existing human-annotated references. Finally, we present some future directions of TS in the era of LLMs.

Problem

• Evaluate LLMs in text simplification tasks
• Compare LLMs with traditional methods
• Assess LLMs across lexical to document levels

Innovation

• Comprehensive LLM performance analysis
• Comparison across four simplification tasks
• LLMs outperform traditional methods significantly