SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

2026-06-28

AI Summary

This study addresses the lack of systematic evaluation of large language models’ (LLMs) reasoning capabilities in software architecture, particularly concerning critical dimensions such as quality attribute trade-offs, design patterns, and system constraints. To bridge this gap, we introduce SAKE—the first standardized benchmark dedicated to software architecture knowledge—comprising 2,154 expert-crafted multiple-choice questions across eight architectural categories and four context lengths. We evaluate 11 prominent LLMs under both zero-shot and five-shot settings. Results reveal generally high overall accuracy but substantial performance disparities across categories, highlighting notable deficiencies in handling specialized architectural tasks. The complete dataset, codebase, and evaluation results are publicly released to foster further research in this domain.

Abstract

Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

Problem

• Software Architecture
• Large Language Models
• Architectural Knowledge
• Benchmark
• Evaluation

Innovation

• Software Architecture
• Large Language Models
• Benchmarking
• Architectural Knowledge
• Evaluation Framework