observability

To address insufficient observability in software systems—leading to slow fault localization and low operational efficiency—this paper designs and implements a lightweight, full-stack observability framework. The framework integrates Java bytecode instrumentation with event stream collection to enable runtime call-chain tracing, performance diagnostics, and root-cause analysis. It introduces a novel dual-mode deployment architecture supporting both online services and on-premises deployment, and achieves cross-toolchain collaborative visualization via tight REST API integration with ExplorViz. Evaluated on the TeaStore benchmark, the system delivers millisecond-scale distributed tracing and real-time heatmap rendering, reduces end-to-end latency by 32%, and shortens mean time to fault identification to the minute level. These results significantly enhance observability and operational intelligence for microservice systems.

Kieker’s observability capabilities are currently limited to a narrow set of languages (e.g., Java, C, Fortran, Python), hindering adoption in modern multi-language systems such as those using C# or JavaScript. To address this gap, we propose the first interoperability framework bridging OpenTelemetry and Kieker. Our approach introduces a distributed tracing data translation middleware that performs semantic mapping and format conversion from OpenTelemetry’s standardized telemetry protocol to Kieker’s event model. This enables unified ingestion of multi-language monitoring data into Kieker’s analysis stack, substantially extending its cross-language observability. Evaluation on the Astronomy Shop benchmark demonstrates accurate reconstruction and visualization of end-to-end call trees, validating both the completeness and practical utility of the transformation. Our work bridges a critical protocol compatibility gap in Kieker’s integration with contemporary observability ecosystems and provides a reusable technical pathway for retrofitting legacy analysis frameworks with emerging open standards.

This study addresses the critical challenge of “silent failures”—erroneous model decisions without system crashes—in production machine learning systems, which undermine conventional monitoring and expose a gap in empirically grounded observability practices. Through seven cross-industry focus group interviews, we applied qualitative thematic coding and scenario mapping to systematically identify the types of observability data practitioners collect and their concrete uses in model validation, anomaly detection, and root-cause diagnosis. Our findings constitute the first empirical characterization of key blind spots in current ML observability tooling: delayed response to feature drift, lack of decision traceability, and difficulty quantifying business impact. Based on these insights, we propose three foundational design principles for next-generation observability tools—explanability-awareness, causal attribution support, and business-impact alignment—and establish an empirically anchored theoretical foundation for future evaluation frameworks and standardization efforts. (149 words)

This work addresses the challenge developers face in comprehensively evaluating the cost, code quality, and behavioral patterns of AI-assisted programming. The paper presents the first AI observability system grounded in real API invocations, integrating precise token tracking, a multi-provider LLM gateway, a pricing database covering 24 models, a response validation pipeline, and LLM-driven code review analysis. A unified dashboard enables joint insights into cost and quality, while Prometheus-based metrics ensure systematic monitoring. In six months of real-world deployment, the system maintained per-review cost estimation errors within 2% and improved the efficiency of analyzing AI usage patterns by an order of magnitude.

In cloud-native microservices, manual and fragmented observability configuration leads to slow fault localization, high resource overhead, and degraded system performance. This paper introduces the first continuous observability assurance methodology, shifting from experience-driven to experiment-driven design. Built upon the Observability eXperimentation (OXN) framework, our approach integrates A/B testing, metric-based feedback loops, and Infrastructure-as-Code (IaC)-enabled automation to dynamically optimize and quantitatively evaluate observability configurations. Evaluated in realistic microservice deployments, our method reduces mean time to detection by 42% on average, decreases sampling overhead by 31%, and—uniquely—enables quantitative validation of how specific observability configurations directly impact Service-Level Objective (SLO) compliance. By establishing a reproducible, iterative, and empirically grounded design paradigm, this work advances observability engineering from ad hoc practice to rigorous, data-driven discipline.

This work addresses the lack of general-purpose, user-friendly architectural support in current software systems for effectively observing sustainability metrics, such as energy consumption. To bridge this gap, the paper proposes the first standardized and modular architectural blueprint specifically designed for sustainability observability, integrating energy monitoring tools and enabling on-demand component configuration and automated deployment. The resulting framework delivers reusable and customizable observability capabilities, which were validated in two real-world scenarios. Empirical results demonstrate that the approach significantly lowers the barrier to implementing sustainability observability, thereby providing practical infrastructure for green software engineering.

Existing approaches to monitoring large language models are confined to isolated layers of the system stack and lack a systematic analysis of the interplay and complementarity among full-stack observability techniques. This work proposes the first unified five-layer AI observability framework, integrating cutting-edge advances from MIT, UC Berkeley, OpenAI, and Microsoft Research in areas such as confidence calibration, internal state probing, chain-of-thought monitoring, cloud operations benchmarking, and non-intrusive tracing. Through a structured evaluation of the strengths, limitations, and applicability of techniques across each layer, the study identifies four critical research gaps and highlights the integration of model-level signals with infrastructure anomalies as the central challenge. This framework lays the theoretical foundation for building end-to-end intelligent operations systems for large language models.

This study addresses the critical gap in observability of code generated by large language models, which, despite being functionally correct, often lacks the instrumentation necessary to expose production-level failures. By deploying 200 microservices on Kubernetes and injecting 13 distinct fault types, this work systematically evaluates the ability of coding agents to recover observability artifacts at the source-code level, revealing a significant disconnect between runtime fault signals and diagnostic semantics embedded in the code. To bridge this gap, the authors propose a skill-guided, observability-oriented approach. Experimental results demonstrate that current agents can only partially reconstruct such artifacts, achieving a maximum fault exposure rate of 13.99%, underscoring the substantial challenge of generating observability code imbued with fault-specific semantic context.

This work addresses a critical limitation in existing observability tools for multi-agent systems, which merely log interactions without enabling real-time enforcement of governance policies, thereby allowing violations to be detected only post hoc. To overcome this, the paper proposes a closed-loop governance architecture that embeds governance attributes directly into the telemetry layer, introducing a Governance-aware Telemetry System (GTS). By integrating an OPA-compatible declarative rule engine, a Governance Execution Bus (GEB), and a trusted telemetry plane, the system achieves real-time policy evaluation and tiered intervention with sub-200-millisecond latency. Leveraging encrypted provenance and enriched telemetry data, GTS establishes a tight feedback loop between observation and action, significantly enhancing runtime compliance and security in multi-agent AI systems.

This work addresses the limitations of traditional Euclidean-space anomaly detection in distinguishing between safety trade-offs and risk accumulation, as well as its inability to adapt to continuously evolving system architectures. Drawing on Rasmussen’s dynamic safety model, the authors propose a drift observability framework in the simplex space, introducing Aitchison geometry and isometric log-ratio coordinates to software system monitoring for the first time. This approach enables coordinate-invariant modeling of compositional operational signals, precisely characterizing drift direction and distance to safety boundaries. By integrating phylogeny-aware aggregation with engineering artifact–driven boundary definitions, the study establishes a continuously comparable mechanism tailored to architectural evolution, facilitating interpretable early warnings and falsifiable hypotheses. The framework thus provides both theoretical foundations and practical pathways for drift observability in complex software systems.