time-to-solution measurement

This study investigates whether the performance gap between large-scale and small, low-cost models will continue to widen or eventually converge amid sustained growth in computational resources. By constructing a classification framework grounded in the functional forms of performance metrics and integrating mathematical modeling, compute scaling laws, and multidimensional capability evaluation, the work systematically analyzes the relationship between training/inference compute and various performance indicators. The findings reveal that bounded metrics inherently favor the widespread adoption of small models, whereas unbounded metrics—particularly those tied to critical capabilities such as software engineering—concentrate high performance among a few resource-rich entities. This research underscores the pivotal role of metric choice in shaping AI development trajectories and policy decisions, while rigorously delineating the conditions under which small models can remain competitive.

Existing AI programming agent benchmarks (e.g., SWE-Bench) focus on isolated, single-step tasks and thus fail to assess long-term software evolution capabilities essential in real-world engineering. To address this gap, we introduce SWE-EVO—the first benchmark explicitly designed for long-horizon software evolution. It comprises 48 multi-step tasks derived from authentic release notes and Git histories of seven open-source Python projects, each involving modifications across an average of 21 files. Tasks emphasize requirement comprehension, cross-file coordination, and functional preservation. We formally define and quantify long-cycle evolution capability, introducing the fine-grained metric *Fix Rate*. Automated validation leverages large-scale test suites (mean 874 tests per instance) and integrates seamlessly with agent frameworks such as OpenHands. Experiments show GPT-5+OpenHands achieves only 21% task success—far below its 65% on SWE-Bench—revealing fundamental limitations in multi-step, multi-file reasoning. SWE-EVO establishes a new standard for evaluating engineering-grade capabilities of AI programming agents.

This work addresses the challenge of automatically translating natural language descriptions of software performance requirements into precise mathematical formulations, a task often hindered by linguistic ambiguity and cognitive uncertainty. The authors propose an interactive, retrieval-augmented preference elicitation method that uniquely integrates domain-specific knowledge into both preference inference and dialogue guidance. By leveraging this knowledge to steer conversational interactions, the approach incrementally refines user intent into accurate mathematical functions. Evaluated on four real-world datasets, the method substantially outperforms ten state-of-the-art baselines, achieving up to a 40-fold improvement in performance with only five rounds of interaction. This significant gain demonstrates its effectiveness in reducing users’ cognitive load while enhancing the efficiency and precision of requirements engineering.

This work addresses the frequent failures in current autonomous AI coding systems—stemming from uncontrolled engineering processes in software and hardware development, particularly in configuration, dependency management, permission handling, and hardware verification. To tackle these challenges, the authors propose the Agentic Agile-V framework, which anchors the development lifecycle in the Agile V-model and introduces a task-level SCOPE-V loop (Specify, Constrain, Orchestrate, Prove, Evolve, Verify) to translate conversational intent into structured engineering artifacts and verifiable evidence. Key contributions include an agent-oriented minimal input artifact taxonomy, a gated mechanism for converting dialogue into formal contracts, a risk-adaptive workflow, and an evidence-bundle-based artifact acceptance model. Empirical results demonstrate that this approach significantly enhances the reliability and controllability of AI-assisted development in complex projects, yielding more stable delivery outcomes and higher verification pass rates.

This work addresses a critical gap in current code agent evaluation frameworks, which typically assume that task specifications are provided fully and upfront, thereby neglecting the iterative, emergent nature of specification refinement inherent in real-world scientific programming. To bridge this gap, the authors introduce the first benchmark tailored to “specification emergence,” comprising 371 verifiable components and approximately 60 rounds of interactive requests. They propose SLUMP (Specification-emergence-based Loss of UMPire fidelity), a novel metric, alongside ProjectGuard—an external state layer designed to enhance specification tracking. Using a five-point fidelity scale and an exposure auditing mechanism, experiments on Claude Code and Codex reveal that incremental specification disclosure substantially degrades model performance. Notably, ProjectGuard recovers 90% of the fidelity gap on Claude Code, increasing fully faithful components from 118 to 181 and reducing severe failures by 23 instances.

This study addresses the trade-offs between inference cost, code quality, and developer experience when deploying enterprise coding agents using either cloud-based APIs or locally quantized models. Through a longitudinal single-developer case study in a real-world production environment, the work presents the first integrated evaluation of Claude Opus (cloud API) and an NVFP4-quantized GLM (local deployment) across multidimensional metrics—including prompt cache efficacy, bug-fixing rate, and total cost of ownership (TCO). Leveraging the Claude Code and Opencode agent frameworks, NVIDIA Blackwell hardware, and a high-hit-rate prompt caching mechanism, the experiments demonstrate that prompt caching reduces the effective API cost to $0.57 per million tokens. While local deployment lowers TCO by 40.1%, it exhibits a 74.9% defect repair rate, substantially increasing debugging overhead and revealing non-trivial engineering trade-offs between cloud and on-premise solutions.

This study addresses the lack of empirical evidence on the long-term impacts of mandatory AI programming tool adoption in enterprise settings, particularly regarding development efficiency and the mediating role of code review mechanisms. Leveraging longitudinal data from a mid-sized, AI-first firm between January 2024 and April 2026—encompassing 802 developers and 196,000 pull requests—the authors employ a staggered difference-in-differences design combined with panel data analysis to uncover the cumulative productivity effects across distinct “adoption” and “usage” phases. By April 2026, the average developer merged 2.09 times more pull requests than at baseline. Automated reviews accounted for over 50% of all reviews, doubling reviewer workload while maintaining stable merge and rollback rates, suggesting that AI has fundamentally reshaped the code review ecosystem.

This work addresses the challenge that state-dependent behaviors in modern computing environments—such as those introduced by adaptive system mechanisms—induce time-dependent biases in traditional software benchmarking, undermining reliable performance comparisons. The paper reframes benchmarking as a decision problem aimed at identifying the fastest program and introduces an experimental paradigm centered on pairwise performance comparisons, thereby avoiding strong assumptions about modeling system dynamics. By leveraging contrastive estimators, consistent statistical inference, and test strategies under finite evaluation budgets, the approach eliminates program-specific biases without relying on absolute performance metrics. The method provides asymptotic guarantees for correct decisions in stateful environments, offering a robust and reliable benchmarking framework for performance-sensitive software development.

Existing evaluations of code-generating agents primarily emphasize functional correctness while overlooking their capacity for performance optimization in real-world scenarios. This work proposes the first end-to-end benchmark tailored to the full performance engineering lifecycle, requiring agents to achieve reproducible performance gains through profiling, diagnosis, code modification, and validation—all while preserving functional correctness. The framework introduces several innovations, including hidden correctness tests, verifiable speedup metrics, and trajectory auditing, integrated with program profiling, cross-layer bottleneck diagnosis, large model–agent collaboration, and an optimization-summary handoff strategy. Evaluation across seven long-horizon tasks and seven agent stacks reveals that optimization efficacy is highly workload-dependent, with no single dominant approach; relying solely on raw speedup ratios can lead to misleading conclusions, necessitating a holistic assessment that jointly considers correctness and reproducibility.

This work addresses two structural failure modes in AI coding agents—context explosion and silent specification-code drift—that undermine development efficiency and code reliability. To tackle these challenges, the paper proposes a lightweight framework that uniquely integrates classical software engineering principles, including information hiding, the C4 model, and Architecture Decision Records (ADRs), into a unified, machine-enforceable architecture. The framework employs a machine-readable specification graph, an ownership-path-based Spine context assembler, a vertical-slice growth protocol, and a drift-gating mechanism to jointly ensure consistency between specifications and generated code. By constraining context expansion and mandating drift detection and correction prior to code merge, the approach significantly enhances the reliability and maintainability of AI-generated code.