MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models

2025-01-25

AI Summary

Large language models (LLMs) exhibit limited capability in understanding Markdown structure. Method: We propose MDEval, the first multilingual benchmark explicitly designed to evaluate Markdown structural awareness, covering Chinese and English across ten academic disciplines with 20,000 high-quality samples. We formally define and quantify the “Markdown Awareness” metric and introduce an interpretable hybrid evaluation paradigm integrating generative tasks with statistical analysis—incorporating multi-dimensional structural parsing, human validation, and Spearman correlation analysis. Contribution/Results: Supervised fine-tuning on MDEval significantly improves open-source LLMs’ Markdown generation quality, achieving structural coherence highly aligned with human judgments (Spearman’s ρ = 0.791; accuracy = 84.1%), approaching GPT-4o performance. The dataset and code are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) are expected to offer structured Markdown responses for the sake of readability in web chatbots (e.g., ChatGPT). Although there are a myriad of metrics to evaluate LLMs, they fail to evaluate the readability from the view of output content structure. To this end, we focus on an overlooked yet important metric -- Markdown Awareness, which directly impacts the readability and structure of the content generated by these language models. In this paper, we introduce MDEval, a comprehensive benchmark to assess Markdown Awareness for LLMs, by constructing a dataset with 20K instances covering 10 subjects in English and Chinese. Unlike traditional model-based evaluations, MDEval provides excellent interpretability by combining model-based generation tasks and statistical methods. Our results demonstrate that MDEval achieves a Spearman correlation of 0.791 and an accuracy of 84.1% with human, outperforming existing methods by a large margin. Extensive experimental results also show that through fine-tuning over our proposed dataset, less performant open-source models are able to achieve comparable performance to GPT-4o in terms of Markdown Awareness. To ensure reproducibility and transparency, MDEval is open sourced at https://github.com/SWUFE-DB-Group/MDEval-Benchmark.

Problem

• Large Language Models
• Markdown Understanding
• Enhancement

Innovation

• MDEval
• Markdown awareness
• language models evaluation