AI-Powered Commit Explorer (APCE)

2025-07-21

AI Summary

Low-quality commit messages are a pervasive issue in version control, impeding code maintainability and knowledge transfer. To address this, we propose an LLM-driven platform for commit message generation and evaluation. Our method comprises three components: (1) a multi-template prompting strategy tailored to diverse code change scenarios; (2) a hybrid evaluation framework integrating automated metrics (e.g., BLEU, ROUGE) with enhanced human evaluation prompts; and (3) an open-source demonstration system supporting real-time generation, comparative analysis, and interactive revision. Empirical evaluation across multiple open-source projects demonstrates significant improvements in functional accuracy and readability of generated messages. The platform exhibits strong effectiveness and scalability, providing both methodological guidance and empirical evidence for the trustworthy adoption of LLMs in software engineering practice.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Commit messages in a version control system provide valuable information for developers regarding code changes in software systems. Commit messages can be the only source of information left for future developers describing what was changed and why. However, writing high-quality commit messages is often neglected in practice. Large Language Model (LLM) generated commit messages have emerged as a way to mitigate this issue. We introduce the AI-Powered Commit Explorer (APCE), a tool to support developers and researchers in the use and study of LLM-generated commit messages. APCE gives researchers the option to store different prompts for LLMs and provides an additional evaluation prompt that can further enhance the commit message provided by LLMs. APCE also provides researchers with a straightforward mechanism for automated and human evaluation of LLM-generated messages. Demo link https://youtu.be/zYrJ9s6sZvo

Problem

• Improving commit message quality using LLMs
• Supporting research on LLM-generated commit messages
• Automating evaluation of AI-generated commit messages

Innovation

• APCE stores diverse prompts for LLMs
• APCE enhances messages with evaluation prompts
• APCE enables automated and human evaluations