Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

2026-06-12

AI Summary

This work addresses the limited understanding of software architecture in current large code models and the high cost and difficulty of manually annotating architectural knowledge. To overcome these challenges, the authors propose a scalable annotation framework based on a dual-agent adjudication mechanism that leverages strong language models (Qwen3-8B/14B/32B) to emulate expert judgment. The framework incorporates two modules—architectural complexity estimation and architectural quality assessment—to automatically generate high-quality, architecture-aware supervision signals without human intervention, enabling large-scale annotation. Evaluated on SWE-bench Verified, the approach achieves a 27.2% resolution rate, representing a 540% improvement over the baseline, and substantially enhances the model’s cross-language generalization and the architectural quality of generated patches.

Abstract

LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation, comprising two judges: the Architecture Complexity Judge (ACJ), which estimates codebase-specific architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates patch conformance to repository-specific architectural conventions via source-grounded rubrics. Fine-tuning Qwen3-8B/14B/32B on 3,360 curated instances achieves resolved rates of up to 27.2% on SWE-bench Verified - up to 540% over the base model and 256% over unfiltered fine-tuning. Meanwhile, the trained models achieve strong cross-language generalization and consistent improvements in architectural patch quality.

Problem

• architectural reasoning
• code LLMs
• scalable labeling
• architectural understanding
• software architecture

Innovation

• agentic judgment
• architectural reasoning
• scalable labeling
• code LLMs
• architecture quality evaluation