DRAFT-ing Architectural Design Decisions using LLMs

2025-04-11

AI Summary

To address the high manual effort and insufficient tool support for documenting Architectural Decision Records (ADRs), as well as the dual challenges of privacy leakage (via third-party LLM APIs) and excessive computational overhead (in self-hosted LLMs), this paper proposes DRAFT: the first domain-specific, retrieval-augmented, few-shot fine-tuned framework for ADR generation. DRAFT innovatively integrates parameter-efficient fine-tuning, retrieval-augmented generation (RAG), and few-shot prompting—constructing a lightweight, domain-adapted model offline and enabling efficient, context-aware generation online via real-time retrieval of relevant examples. Evaluated on a large-scale, real-world dataset of 4,911 ADRs, DRAFT significantly outperforms state-of-the-art baselines across both automated metrics (BLEU-4, ROUGE-L) and human evaluations (relevance, completeness, readability), achieving a balanced trade-off among generation quality, data privacy, and inference efficiency.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Architectural Knowledge Management (AKM) is crucial for software development but remains challenging due to the lack of standardization and high manual effort. Architecture Decision Records (ADRs) provide a structured approach to capture Architecture Design Decisions (ADDs), but their adoption is limited due to the manual effort involved and insufficient tool support. Our previous work has shown that Large Language Models (LLMs) can assist in generating ADDs. However, simply prompting the LLM does not produce quality ADDs. Moreover, using third-party LLMs raises privacy concerns, while self-hosting them poses resource challenges. To this end, we experimented with different approaches like few-shot, retrieval-augmented generation (RAG) and fine-tuning to enhance LLM's ability to generate ADDs. Our results show that both techniques improve effectiveness. Building on this, we propose Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, which combines the strengths of all these three approaches for more effective ADD generation. DRAFT operates in two phases: an offline phase that fine-tunes an LLM on generating ADDs augmented with retrieved examples and an online phase that generates ADDs by leveraging retrieved ADRs and the fine-tuned model. We evaluated DRAFT against existing approaches on a dataset of 4,911 ADRs and various LLMs and analyzed them using automated metrics and human evaluations. Results show DRAFT outperforms all other approaches in effectiveness while maintaining efficiency. Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.

Problem

• Standardizing Architectural Knowledge Management is challenging
• Manual effort limits Architecture Decision Records adoption
• LLMs need enhancement for quality ADD generation

Innovation

• Combines few-shot, RAG, and fine-tuning techniques
• Uses two-phase offline and online processing
• Outperforms existing methods in effectiveness