SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works

2025-05-23

AI Summary

Existing release note generation tools suffer from poor domain adaptability, insufficient audience awareness, high integration overhead, and limited practicality. This paper proposes the first end-to-end large language model (LLM)-based framework specifically designed for release note generation. Our approach innovatively fuses multi-source contextual signals—including code diffs, commit messages, and pull request metadata—to enable semantic-driven dynamic classification and importance scoring, thereby supporting personalized, plug-and-play, structured summary generation. Compared against baselines such as DeepRelease, our method achieves state-of-the-art performance across four key dimensions: completeness, clarity, conciseness, and organization. Human evaluation ranks it first in completeness, organization, and clarity; automated metrics yield the highest composite score. Extensive experiments validate its strong contextual awareness and cross-project generalizability.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The release note is a crucial document outlining changes in new software versions. Yet, many developers view the process of writing software release notes as a tedious and dreadful task. Consequently, numerous tools have been developed by researchers and practitioners to automate the generation of software release notes. However, these tools fail to consider project domain and target audience for personalisation, limiting their relevance and conciseness. Additionally, they suffer from limited applicability, often necessitating significant workflow adjustments and adoption efforts, hindering practical use and stressing developers. Despite recent advancements in natural language processing and the proven capabilities of large language models in various code and text-related tasks, there are no existing studies investigating the integration and utilisation of LLMs in automated release note generation. Therefore, we propose SmartNote, a novel and widely applicable release note generation approach that produces high-quality, contextually personalised release notes using LLM technology. SmartNote aggregates changes and uses an LLM to describe and summarise the changes using code, commit, and pull request details. It categorises and scores commits to generate structured and concise release notes of prioritised changes. Our human and automatic evaluations reveal that SmartNote outperforms or achieves comparable performance to DeepRelease, Conventional Changelog, and the projects'original release notes across four quality metrics: completeness, clarity, conciseness, and organisation. In both evaluations, SmartNote ranked first for completeness and organisation, while clarity ranked first in the human evaluation. A further evaluation demonstrates that SmartNote is effective in terms of context awareness and applicability.

Problem

• Automates personalized software release note generation using LLMs
• Addresses lack of domain and audience awareness in existing tools
• Improves relevance and conciseness without workflow disruption

Innovation

• Uses LLM for personalized release notes
• Aggregates and scores commits automatically
• Outperforms existing tools in quality metrics