Controllable Abstraction in Summary Generation for Large Language Models via Prompt Engineering

2025-10-17

AI Summary

Large language models (LLMs) exhibit unstable summarization quality and limited controllability over abstraction levels. Method: This paper proposes a controllable abstractive summarization framework based on multi-stage prompt engineering, integrating semantic analysis, topic modeling, and noise-aware control to enable adjustable abstraction granularity. We systematically investigate the impact of prompt length, data noise, and text genre on summarization performance using the CNN/Daily Mail benchmark. Contribution/Results: Experiments demonstrate that medium-length prompts yield statistically significant improvements in ROUGE-L scores; increased input noise degrades performance consistently; and LLMs generalize best on news-domain texts. The framework provides an interpretable, configurable pathway to enhance accuracy, consistency, and abstraction-level control in LLM-generated summaries.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

This study presents a controllable abstract summary generation method for large language models based on prompt engineering. To address the issues of summary quality and controllability in traditional methods, we design a multi-stage prompt generation framework. This framework generates summaries with varying levels of abstraction by performing semantic analysis, topic modeling, and noise control on the input text. The experiment uses the CNN/Daily Mail dataset and provides a detailed analysis of different prompt lengths, data noise, and text types. The experimental results show that prompt length has a significant impact on the quality of generated summaries. Both very short and very long prompt tokens result in a decrease in summary quality. Data noise also negatively affects the summary generation process. As noise levels increase, the ROUGE-L score gradually decreases. Furthermore, different text types have varying effects on the model's ability to generate summaries. The model performs best when handling news texts, while its performance is worse when processing academic articles. This research provides new insights into improving summary generation using large language models, particularly in how controlling prompt strategies and optimizing text preprocessing can enhance summary accuracy and controllability.

Problem

• Enhancing summary quality and controllability in large language models
• Optimizing prompt length to improve abstractive summarization performance
• Mitigating negative effects of data noise on generated summaries

Innovation

• Multi-stage prompt framework controls abstraction levels
• Semantic analysis and topic modeling enhance summary quality
• Prompt length optimization improves text generation accuracy