CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System

2026-06-17

AI Summary

This work addresses the challenge of automatically assessing structural integrity and requirements traceability in software architecture assignments, which existing methods struggle to evaluate accurately. The authors propose a multi-agent large language model system that integrates specialized agents with Python microservices to collaboratively parse textual descriptions and UML diagrams, generating personalized LaTeX feedback conforming to predefined templates. A novel deterministic evidence anchoring mechanism based on normalized Levenshtein distance is introduced, alongside a ConsistencyManager agent that performs cross-validation, deduplication, and result fusion. Multimodal document extraction leverages PyMuPDF and a vision-enhanced LLM (GPT-4o). Evaluated on ten student reports, the system satisfies 88.8% of assessment criteria, demonstrates moderate agreement with human graders (Cohen’s κ = 0.582), and processes each report in approximately four minutes.

Abstract

Automated assessment in software engineering education has advanced significantly for code grading and essay scoring. However, reviewing software architecture deliverables, which requires analyzing structural completeness and requirements traceability, has not yet been fully automated. Applying Large Language Models (LLMs) to this task requires robust architectures to ensure technical feedback is accurate and reliable for students. This paper presents CAPRA (Configurable Architecture Proficiency Report Assessment), a multi-agent LLM system that analyzes software architecture deliverables to generate personalized, template-compliant LaTeX feedback. As a core design choice, CAPRA coordinates multiple specialized agents and employs a Python-based microservice for multi-modal document extraction, utilizing PyMuPDF and vision-enabled LLMs (specifically gpt-4o) to parse text and UML diagrams. To ensure educational reliability and mitigate hallucinations, CAPRA introduces a deterministic Evidence Anchoring step using fuzzy matching via normalized Levenshtein distance, along with a ConsistencyManager agent that cross-verifies, deduplicates, and merges findings. System performance is assessed using a structured eight-criterion binary evaluation taxonomy covering: (i) extraction completeness, (ii) feature validation, (iii) issue grounding and severity detection, (iv) recommendation specificity and traceability, and (v) template and tone compliance. A preliminary empirical evaluation on 10 student reports shows that CAPRA satisfied 88.8% of the evaluated criteria under a strict two-rater aggregation rule, achieved moderate inter-rater agreement with human evaluators (kappa = 0.582), and processed each report in slightly over 4 minutes. While these results support the viability of LLM-supported architectural feedback, human oversight remains essential for subjective assessment dimensions.

Problem

• software architecture assessment
• automated feedback
• requirements traceability
• structural completeness
• LLM-based evaluation

Innovation

• multi-agent LLM
• software architecture assessment
• evidence anchoring
• multimodal document extraction
• automated feedback