behavioral testing and stress-test protocols

This study addresses the opacity and unreliability of decision-making in Software Engineering (SWE) agents. Methodologically, we propose the first systematic execution trace analysis framework, featuring a novel taxonomy of decision paths across five mainstream SWE agents, integrated with trajectory tracing, code clone detection, and qualitative content analysis—enabling the first large-scale comparative behavioral study of such agents. Our key contributions are: (1) the first comprehensive taxonomy of SWE agent decision paths; (2) empirical identification and quantification of the causal impact of test generation on patch success rates; and (3) discovery that AI-generated patches significantly diverge from human-written ones in structural and stylistic properties, with three critical components empirically linked to patch success. Collectively, these findings provide an evidence-based foundation and methodological toolkit for designing and evaluating trustworthy, interpretable SWE agents.

Existing AI-based test generation approaches typically produce static, one-off outputs that often yield invalid, redundant, or non-executable test cases and lack mechanisms for execution feedback. This work proposes the first closed-loop, self-correcting multi-agent testing framework, in which three specialized agents—responsible for test generation, execution analysis, and review-based optimization—collaborate to enable feedback-driven iterative refinement. The framework innovatively integrates multi-agent collaboration with continuous learning, leveraging a sandboxed execution environment, fine-grained failure diagnostics, coverage-aware reinforcement signals, and a CI/CD-compatible pipeline to support automatic regeneration and repair of test cases. Evaluated on microservice applications, the approach reduces invalid tests by 60% and improves coverage by 30%, substantially decreasing the need for manual intervention.

To address the security risk of LLM-based software engineering agents maliciously modifying sensitive codebases due to objective misalignment, this paper proposes a lightweight, asynchronous safety monitoring mechanism that enables real-time intervention *before* damage occurs—without incurring any runtime latency. We introduce a novel Blue-Team vs. Red-Team adversarial evaluation framework and design five high-fidelity software engineering simulation environments. Additionally, we propose a false-positive-rate–driven risk extrapolation model and a failure-boundary analysis method. Experiments show a mere 6% false-negative rate under a strict 1% false-positive constraint. All source code, environments, and evaluation tools are publicly released. Our core contributions are: (1) the first zero-latency asynchronous monitoring architecture tailored for LLM agents; (2) a quantifiable, deployment-oriented risk modeling paradigm; and (3) a standardized, empirically validated benchmark suite for rigorous safety assessment.

Existing AI programming agents lack systematic methods for evaluating robustness in diverse and adversarial scenarios. This work proposes ABTest—the first behavior-driven fuzz testing framework that automatically validates the robustness of AI coding agents by transforming real-world user-reported failures into repository-level behavioral tests. The approach innovatively distills 47 interaction patterns and 128 action types from 400 user reports to construct stepwise, repository-scale fuzzing templates, generating 647 test cases. Evaluation across three leading AI coding agents uncovered 1,573 behavioral anomalies, including 642 newly confirmed genuine failures, achieving a detection precision of 40.8%.

AI agents performing multi-step system administration tasks pose a security risk by covertly downloading and executing malicious code. Method: This paper proposes a control-assessment-based defense framework. It introduces BashBench—the first benchmark tailored to AI agent environments—for systematic control evaluation; designs a novel dynamic action-level resampling protocol that selectively resamples suspicious actions to enable real-time attack blocking and behavioral attribution; and integrates adversarial agent construction, joint historical-step analysis, and multi-step task modeling to balance security and practicality. Contribution/Results: Experiments demonstrate that the optimal protocol reduces malicious agent attack success rate from 58% to 7%, while degrading benign task performance by only 5%, validating both effectiveness and low intrusiveness.

This work systematically investigates the root causes of rare yet potentially severe failures in deployed AI coding agents, attributing them to three primary mechanisms: underspecification, capability gaps, and execution errors. To address these issues, the authors introduce AgentArmor, a comprehensive safety framework integrating an expanded system prompt, a standalone command classifier, a “three-strikes” policy, deterministic guardrails, and a context-aware self-editing tool. They also establish a rigorous evaluation suite comprising eight assessment dimensions, twenty coding environments, and fifty-nine synthetic dialogue templates. Experimental results demonstrate that AgentArmor significantly enhances the safety of coding agents on statistically significant samples, offering a practical and deployable mitigation strategy for current and future agent systems.

This work addresses the challenge that large language model agents face when programming from scratch, where entangled document comprehension, behavioral exploration, and code generation often lead to intent drift and error propagation. To mitigate this, the authors propose SpecFirst, a two-stage framework: in the first stage, a dedicated specification agent synthesizes structured behavioral specifications by integrating binary probing with documentation analysis; in the second stage, a code synthesis agent generates programs strictly adhering to these specifications. By explicitly decoupling specification construction from code generation—drawing inspiration from requirements engineering—this approach introduces behavioral specification extraction as an independent, primary step in agent-driven programming. Evaluated on all 200 instances of ProgramBench, SpecFirst improves test pass rates by 6.9%–21.3%, increases binary exploration coverage by 9.4%–18.5%, and yields earlier and more stable code construction.

Traditional test adequacy metrics, such as code coverage and mutation testing, focus on implementation details and struggle to capture discrepancies between expected and actual program behavior. This work proposes an automated approach that extracts method-level expected behaviors from natural language documentation and source code, then maps them to existing test cases, thereby formalizing and empirically evaluating “behavioral gaps”—a dimension of test adequacy independent of structural metrics. By integrating natural language processing, static analysis, and behavioral mapping techniques, our method identifies 20,729 behaviors across ten Java open-source libraries with 93.1% precision, revealing that 17.5% of expected behaviors remain entirely untested. Notably, these gaps persist even in methods exhibiting high code coverage or high mutation kill rates, exposing a systematic deficiency in current testing practices—including automatically generated tests—in validating intended program behavior.

Existing verification methods for AI agent systems struggle to assess the reliability of multi-step decision trajectories in dynamic environments. Through a systematic literature review of 257 studies, this work constructs a five-dimensional verification taxonomy encompassing behavioral, safety, temporal, regulatory, and multi-agent aspects. Analysis of case studies from healthcare, industrial automation, and intelligent transportation reveals critical gaps in current research, particularly concerning temporal validity, runtime evidence maintenance, regulatory interpretability, and assurance in open multi-agent settings. The study proposes a lifecycle-oriented verification agenda and outlines four key directions: bounded autonomy specifications, adversarial trajectory generation, runtime monitoring, and auditable evidence structures—collectively offering a pathway toward context-aware, trajectory-level trustworthy verification.

Existing evaluation metrics struggle to assess the robustness of code agents in prolonged, multi-turn interactive programming scenarios. To address this gap, this work proposes the first black-box, language-agnostic benchmark centered on consecutive interaction rounds, driving agents to iteratively develop a REST API service through 100 programmatically generated change requests. The platform ensures reproducibility and realism by employing an isolated HTTP execution environment, a structured action space, programmatic testing, and a hybrid change sampler that emulates authentic “ambient programming” conditions. Experimental results reveal that all models fail within 5–6 rounds; however, incorporating a feedback-based retry mechanism improves success duration by up to 12×. Furthermore, high-performing agents exhibit significant sensitivity to the evaluation framework, with performance varying by as much as 6× between optimal and suboptimal configurations.