resilience testing

Safety-critical cyber-physical systems (CPS) rely on assurance cases to substantiate safety claims, yet existing approaches lack systematic, empirically grounded methods for identifying and classifying defeaters—counters to the validity or sufficiency of assurance arguments. Method: Drawing on a 20-year systematic literature review and open coding of real-world assurance cases, this work proposes the first empirically derived, seven-dimensional defeater taxonomy. Contribution/Results: The taxonomy categorizes defeaters along seven dimensions—including insufficient evidence, broken inference chains, and ambiguous claims—thereby establishing the first standardized framework for defect analysis in assurance cases. We release an open-source, reusable defeater classification ontology and supporting toolkit, enabling consistent interpretation, reproducible evaluation, and scalable management of defeaters. This provides the first benchmark framework explicitly designed to enhance the credibility and trustworthiness of safety arguments for both industry and academia.

This paper addresses the under-recognized reliability challenges of large language models (LLMs) in real-world system deployments. Adopting a systems engineering perspective, it establishes the first fault taxonomy for LLM-based applications. Through systematic analysis and multi-case root-cause investigation, the study identifies 15 classes of latent failures—including multi-step reasoning drift, context boundary degradation, erroneous tool invocation, and latent inconsistency—exposing fundamental limitations of current evaluation benchmarks in stability, reproducibility, and workflow integration. The work introduces high-level design principles centered on observability, cost sensitivity, and version evolution, shifting LLM reliability research from a model-centric to a system-integration paradigm. It delivers the first structured fault classification framework and practical guidance for building reliable, maintainable, and auditable LLM software systems.

Chaos engineering lacks a systematic, comprehensive review in the literature. Method: This paper conducts the first multi-source literature review (MLR), systematically analyzing 96 academic and gray literature sources published between 2016 and 2024—including 88 core publications from 2019 to 2024. It synthesizes findings via thematic clustering and qualitative coding. Contribution/Results: The study establishes the first consensus definition of chaos engineering, proposes a four-layer capability model and a five-dimensional component taxonomy, and performs a cross-tool evaluation of 12 mainstream chaos engineering tools. It identifies six open research challenges and clarifies practice drivers, tool characteristics, and research evolution trends. The results provide a foundational theoretical framework, methodological benchmark, and roadmap for future work—bridging critical knowledge gaps between academia and industry.

Traditional Failure Mode and Effects Analysis (FMEA) development for industrial equipment is highly manual, time-consuming, and suffers from low knowledge reuse. Method: This paper proposes a foundation-model-based approach for automated FMEA generation and structured database ingestion. It integrates domain-adapted natural language processing and information extraction to accurately identify fault modes, effects, causes, and detection mechanisms from unstructured technical documents, mapping them to standardized FMEA table entries. An interpretable, interactive correction mechanism enables expert feedback integration for iterative refinement, while structured outputs are automatically persisted into a relational database. Contribution/Results: Experiments demonstrate over 80% reduction in FMEA development cycle time, significantly improving efficiency and consistency in industrial asset knowledge construction. The approach validates the feasibility and practical value of foundation models in high-reliability industrial knowledge engineering applications.

This study addresses the lack of structured mechanisms for mitigating systemic risks—legal, reputational, and financial—arising from failures of high-risk AI systems. Leveraging 6,893 incident reports extracted from a corpus of 9,705 media articles, the authors employ structured prompt engineering to identify and categorize mitigation strategies. They extend MIT’s existing AI risk mitigation taxonomy by introducing four new strategic categories: “Correction and Containment,” “Legal/Regulatory and Enforcement,” “Financial and Market Controls,” and “Avoidance and Denial,” which collectively encompass 67% of newly identified subcategories. The resulting dataset comprises 32 distinct mitigation labels, supported by 23,994 annotations—including 9,629 instances of previously undocumented patterns—thereby substantially enhancing diagnostic and intervention capabilities for emerging systemic failures and improving post-deployment monitoring efficacy.

This study addresses a fundamental structural incompatibility between iCloud Drive’s file system interface and its underlying cloud synchronization mechanism, which arises from the erroneous mapping of distributed causal relationships onto a linear timeline. This misalignment induces data inconsistency and corruption when iCloud Drive interoperates with tools such as Time Machine and Git. The paper identifies this class of failures as a “category error” and establishes its deep connection to the impossibility of achieving consistency under network partitions. Drawing on distributed systems theory, causal graph modeling, and empirical analysis of 366 GB of divergent state data, the work delineates five root causes of conflict. It further demonstrates that the Open Atomic Ethernet (OAE) transactional semantics framework offers a structurally sound solution aligned with physical reality.

This study addresses the multidimensional risks—operational, security, and governance-related—that enterprises face when deploying large language models, noting that existing open-source tools are fragmented and fail to comprehensively cover authoritative risk taxonomies. To bridge this gap, the work proposes a structured mapping protocol that automatically aligns the capabilities of 21 prominent open-source tools with the 32 subcategories of the MIT AI Risk Framework, leveraging retrieval-augmented generation (RAG) and LLM-based parsing. The protocol’s validity is substantiated through source code and documentation analysis, majority voting, and inter-rater reliability assessment using Fleiss’ Kappa (κ = 0.509, F1 = 75.5%). Findings reveal a pronounced overconcentration of current tools on technical controls, with significant gaps in governance, legal, and market risk domains, thereby providing an empirical foundation for developing layered AI risk mitigation architectures.

Current agent evaluation practices often reduce failures to system-level outcomes, making it difficult to pinpoint root causes or guide effective remediation. This work proposes an interaction-centric failure taxonomy and introduces, for the first time, a cross-architectural and generalizable framework for failure localization. The framework maps 41 distinct failure modes onto interaction edges between components—such as models, toolchains, and environments—and explicitly delineates responsibility boundaries among them. By integrating component interaction graph attribution, multi-source trajectory analysis, and an independent reasoning agent-based evaluator, the approach enables reproducible validation. Experiments across four state-of-the-art models demonstrate that the strongest evaluator achieves a Cohen’s κ of 0.76 with human annotations, confirming the taxonomy’s generalizability and consensus alignment.

This study addresses key challenges in reliability modeling of cyber-physical systems (CPS), including heavy reliance on expert knowledge, incomplete failure documentation, and inadequate representation of subsystem interactions. To overcome these limitations, the authors propose a Capability Interaction Graph (CIG) grounded in the Unified Foundational Ontology (UFO) to construct a semantic knowledge graph for CPS. This framework automatically derives fault trees to identify failure propagation paths and minimal cut sets. By integrating an ontology-driven CIG with knowledge graph technology, the approach enables unified semantic modeling across engineering domains and supports automated fault tree generation. Experimental results demonstrate that the proposed method explicitly captures functional dependencies and system semantics, significantly reducing modeling complexity while enhancing the accuracy of fault analysis.

This study addresses the challenge of effectively monitoring early-stage agent systems, where structural flaws often obscure task-level errors. The authors propose a three-dimensional (quality, suitability, efficiency) and three-granularity (intra-run, inter-run, structural) monitoring and triaging framework tailored for low-maturity agent systems. They introduce a novel system maturity staging model based on the coefficient of variation and monitoring granularity, integrated with a severity classification adapted from FMEA to guide human review. The resulting transferable monitoring architecture supports document-driven, multi-stage workflows, enhanced by a synthetic testbed with controlled error injection. Experimental results demonstrate that structural defects significantly mask task-level signals; 97% of issues can be automatically traced, with only 2% requiring human intervention, and each granularity level precisely identifies its corresponding defect type (coefficients of variation: 0.02, 1.25, and 0.00, respectively).

This work establishes that large language models cannot achieve universal reliability across all tasks due to the infinitely diverse failure modes they encounter, which cannot be fully mitigated by any finite set of interventions. For the first time, it formally proves the impossibility of全域 (universal) reliability and advocates shifting focus to locally tractable “patch domains”—such as legal review or medical question-answering—where error modes are sparse and enumerable. By constructing a taxonomy of failure modes, an intervention dictionary, and an active mode exposure model, the theory demonstrates that the required intervention budget grows only polylogarithmically with sequence length and asymptotically approaches a constant once the catalog of failure modes saturates. Empirical results confirm that this approach substantially reduces residual error rates within bounded task domains, offering both a theoretical foundation and a practical pathway toward high-reliability deployment.

This work addresses the limitations of generic pretrained models in industrial-scale video and live-stream content moderation, where platform-specific data distributions, policy objectives, and safety constraints are poorly aligned with off-the-shelf solutions, and systematic failure diagnosis and remediation mechanisms are lacking. The paper introduces a diagnostic methodology for audio-visual language models (AVLMs) that pioneers the characterization of model failures through observable feature signatures and establishes a principled mapping between failure categories and targeted intervention strategies, replacing heuristic trial-and-error approaches. Built upon multimodal foundation model architectures and validated on real-world platform traffic, this framework enables precise interventions throughout the model lifecycle. The resulting AVLM system has been deployed across more than 100 regions globally, significantly improving moderation accuracy and traceability for high-noise, semantically ambiguous, and highly diverse content.