Development of Automated Software Design Document Review Methods Using Large Language Models

2025-09-12

AI Summary

This study addresses the low efficiency and high cost of manual review for software design documentation by proposing an LLM-based automation framework for multi-perspective review. Methodologically, it systematically categorizes eleven review dimensions and identifies those amenable to automation using general-purpose large language models (e.g., GPT); for hybrid semi-structured documents containing tables, it integrates prompt engineering with context-aware modeling to enhance the model’s comprehension of design logic and cross-document consistency. The key contribution is overcoming current LLM limitations in interpreting structured information, enabling— for the first time—the automated detection of cross-document design inconsistencies across multiple perspectives. Experimental evaluation on real-world industrial documentation demonstrates that the approach accurately identifies inconsistency issues, achieving an accuracy level meeting industrial deployment requirements.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this study, we explored an approach to automate the review process of software design documents by using LLM. We first analyzed the review methods of design documents and organized 11 review perspectives. Additionally, we analyzed the issues of utilizing LLMs for these 11 review perspectives and determined which perspectives can be reviewed by current general-purpose LLMs instead of humans. For the reviewable perspectives, we specifically developed new techniques to enable LLMs to comprehend complex design documents that include table data. For evaluation, we conducted experiments using GPT to assess the consistency of design items and descriptions across different design documents in the design process used in actual business operations. Our results confirmed that LLMs can be utilized to identify inconsistencies in software design documents during the review process.

Problem

• Automating software design document review using LLMs
• Identifying which review perspectives LLMs can handle
• Developing techniques for LLMs to comprehend complex design documents

Innovation

• Automated review using LLM techniques
• Developed methods for table data comprehension
• GPT evaluates design consistency across documents