Evaluating and Improving Graph to Text Generation with Large Language Models

2025-01-24

AI Summary

This paper addresses the weak planning capability and insufficient factual grounding of large language models (LLMs) in complex network graph-to-text generation. Methodologically, it (1) introduces PlanGTG—the first graph-to-text benchmark explicitly designed to evaluate planning ability, featuring two subtasks: node reordering and attribute attribution; (2) proposes a diversity-and-difficulty-aware few-shot example selection strategy; and (3) integrates graph-structured serialization, few-shot prompting, supervised fine-tuning, and inference-time optimization. Experiments on PlanGTG demonstrate substantial improvements in factual accuracy, coherence, and structural fidelity of generated text. Both few-shot and fine-tuned variants achieve breakthrough performance across human evaluation and automated metrics—including FactScore and BLEURT—outperforming prior approaches. This work establishes a novel paradigm for graph reasoning and controllable text generation grounded in structured planning and factual consistency.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Generation
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) have demonstrated immense potential across various tasks. However, research for exploring and improving the capabilities of LLMs in interpreting graph structures remains limited. To address this gap, we conduct a comprehensive evaluation of prompting current open-source LLMs on graph-to-text generation tasks. Although we explored the optimal prompting strategies and proposed a novel and effective diversity-difficulty-based few-shot sample selection method, we found that the improvements from tuning-free approaches were incremental, as LLMs struggle with planning on complex graphs, particularly those with a larger number of triplets. To further improve LLMs in planning with graph sequences and grounding in truth, we introduce a new graph-to-text dataset, PlanGTG, annotated with two sub-tasks: reordering and attribution. Through extensive automatic and human evaluations, we demonstrate significant improvements in the quality of generated text from both few-shot learning and fine-tuning perspectives using the PlanGTG dataset. Our study paves the way for new research directions in graph-to-text generation. PlanGTG datasets can be found in https://github.com/probe2/kg_text.

Problem

• Large Language Models
• Complex Network Graphs
• Textual Description Generation

Innovation

• Large-scale Language Models
• Graph-to-Text Generation
• PlanGTG Dataset