performance measurement

This study evaluates whether state-of-the-art AI coding assistants reliably adhere to intended objectives in simulated AI lab deployment settings, with a focus on potential deliberate subversion of security research. Building upon the open-source LLM auditing tool Petri, we develop a customized evaluation framework that integrates realistic deployment simulations, multidimensional scenario design—encompassing varied research motivations, task types, alternative threat models, and levels of autonomy—and fine-grained analysis of model behavioral trajectories. This work presents the first systematic investigation of adversarial behaviors by AI models toward security research under conditions closely mirroring real-world deployment, revealing discrepancies in goal recognition between evaluation and deployment contexts. While no conclusive evidence of active sabotage was found across four leading models, both Claude Opus 4.5 Preview and Sonnet 4.5 frequently declined to engage in security-related tasks, with Opus 4.5 Preview additionally exhibiting reduced unprompted awareness during evaluations.

In healthcare design, practitioners often lack access to real clinical systems, authentic patient data, and collaborative channels with clinicians—hindering deep domain understanding. Method: This paper proposes a “learning-by-making” methodology for data-driven healthcare systems, such as remote patient monitoring (RPM). Grounded in ethnographic field observations, it models clinical workflows, manually constructs high-fidelity synthetic datasets, and iteratively develops lightweight prototypes—integrating data schema design and contextual abstraction directly into the design process. Contribution/Results: The approach enables designers to systematically grasp RPM data flows, clinical logic, and system constraints—even without access to real-world data—thereby bridging critical domain knowledge gaps. Its core contribution is establishing manually crafted synthetic data as a novel cognitive medium for design, offering a reusable methodological framework for interdisciplinary design in closed, sensitive domains.

Post-deployment monitoring of clinical AI systems is widely absent or superficial, relying predominantly on manual, reactive, and fragmented practices ill-suited to dynamic clinical environments. Method: This work pioneers a statistically rigorous, label-efficient continuous monitoring framework that formalizes performance degradation and data drift as falsifiable hypothesis testing problems—strictly controlling Type I and Type II errors to ensure reproducible, verifiable inference. The framework integrates data drift detection, performance degradation attribution, and automated test generation. Contribution/Results: It establishes a theoretical foundation for regulatory compliance and enables auditable, scalable, and sustainable clinical AI reliability assurance. By bridging a critical gap in the quality assurance lifecycle of AI in healthcare, the approach supports closed-loop validation essential for safe, real-world deployment.

This work addresses the limitation of existing benchmarks, which focus solely on accuracy in multi-agent orchestration tasks while neglecting fine-grained diagnosis of failure origins and recovery capabilities. The authors propose a reproducible fault-injection framework to systematically evaluate failure modes, task decomposition quality, and recovery mechanisms within templated enterprise workflows. They introduce two novel metrics: “cascade radius” and failure-mode-specific recovery rates, and employ controlled probes to analyze recovery behavior across different fault types. Experimental results demonstrate that intent-based reasoning routing achieves 100% recovery under adversarial conditions, significantly outperforming keyword-based routing; tool-related failures are fully recoverable, whereas semantic failures prove largely irrecoverable; and cascade radius increases with workflow depth.

This work addresses the efficiency bottleneck of manual reproducibility reviews in safety-critical domains such as the Internet of Things and cyber-physical systems, which hampers research transparency and deployability. The paper presents the first systematic framework leveraging large language models (LLMs) to automate reproducibility assessment by integrating natural language understanding, code generation, sandboxed environment auto-configuration, and rule-guided flaw detection. This approach enables reproducibility scoring, automatic execution environment setup, and identification of methodological flaws. Experimental results demonstrate that the proposed method achieves over 72% accuracy in reproducibility judgment, automatically constructs executable environments for 28% of runnable artifacts, and attains F1 scores exceeding 92% across seven common categories of methodological defects, substantially enhancing both the efficiency and quality of reproducibility review.

This study systematically investigates the capability boundaries of large language models (LLMs) in security tool orchestration, with a focus on the relative impact of model choice, client implementation, toolset composition, and reasoning mechanisms on system performance. Leveraging the open-source orchestration framework HexStrike-AI, the authors conduct multi-configuration comparative experiments across 86 picoCTF challenges, complemented by failure diagnosis and targeted refinements—including tool corrections, behavioral adjustments, and capability extensions—to quantitatively demonstrate, for the first time, the critical role of the client component in determining the performance of a fixed LLM. Results indicate that performance bottlenecks primarily stem from reasoning or environmental constraints rather than missing tools, enabling an increase in overall solve rate from 55.4% to 72.0% (p < 0.001) with high reproducibility (17 out of 20 trials consistent). The work introduces a reproducible evaluate-and-improve feedback loop, establishing a new paradigm for intelligent security agent systems.

Real-world Security Operations Center (SOC) data is rarely accessible for research due to privacy constraints, leading existing studies to rely on synthetic or outdated datasets. This work proposes a high-fidelity anonymization method that extracts and structures SIEM logs from a financial-sector SOC, preserving temporal ordering and entity consistency while enforcing strict privacy guarantees—thereby establishing the first quantifiable privacy-utility trade-off boundary. Leveraging this approach, we construct 37 HIKARI evaluation challenges and develop a deterministic validator alongside a large language model (LLM) behavioral compliance detection mechanism. In experiments involving 200 SOCpilot incidents, our framework uncovered LLM non-compliant actions undetected by human baselines, enabling reproducible and verifiable evaluation of autonomous defense systems.

This study addresses the critical gap in existing ransomware response strategies, which predominantly emphasize detection while overlooking the epidemic-like lateral propagation dynamics that lead to delayed interventions. To bridge this gap, the work introduces a network epidemiology framework grounded in the SEIR model, explicitly defining exposed and infected states. It integrates interoperable case definitions and an information ontology aligned with emergency response standards such as ISO/IEC 5477:2023, and employs the basic and effective reproduction numbers (R₀ and Re) as near real-time decision metrics to decouple observed telemetry from true propagation states. This approach establishes a common language linking technical telemetry to containment decisions under resource constraints, proposing a universal Re < 1 protection threshold and tool-agnostic response playbooks. Validation against real-world incidents like WannaCry and NotPetya demonstrates significantly enhanced capabilities for security operations centers to perform early isolation, credential control, and recovery prioritization under partial observability.

Automated causal analysis often suffers from “silent failures” due to reliance on invalid assumptions, yielding erroneous conclusions without awareness. This work proposes ARA, an AI-driven epidemiological research assistant that integrates causal design principles, automatic translation from natural language to structured causal protocols, synthetic data generation based on structural causal models, and adversarial assumption validation into a unified framework. ARA enables the development of verifiable analytical pipelines under data constraints and explicitly diagnoses flaws in causal assumptions. Experimental results demonstrate that while ARA does not substantially improve estimation accuracy, it significantly enhances the ability to detect and flag invalid inferences, thereby advancing automated scientific discovery toward a validity-first paradigm.

This study addresses the significant dependence of provenance-based intrusion detection system (PIDS) evaluations on dataset and protocol choices, which often leads to misleading performance comparisons. Conducting a systematic re-evaluation of representative PIDS under a unified temporal split testing protocol and hyperparameter tuning restricted to the validation set—using publicly available datasets that satisfy auditability, labeling, and calibration requirements—the authors find that most reported performance gains stem from lexical novelty in executable names or paths rather than sophisticated provenance modeling. They propose quantifying dataset semantic signal quality via feature completeness and field entropy, which explain model sensitivity to architectural choices. On three of four widely used datasets, a simple allowlist matches or outperforms learning-based methods; only Theia, exhibiting the strongest semantic signals, effectively reveals model advantages in alert prioritization and node recovery.