performance metrics

The absence of standardized frameworks for data valuation and monetization impedes organizations’ systematic assessment and realization of data value. Method: Through a systematic literature review (N=162) integrating thematic analysis and taxonomy development, this study proposes— for the first time—the Balanced Scorecard–based Comprehensive Data Valuation and Monetization Framework. Contribution/Results: The framework spans strategic, business, technical, and governance dimensions, offering a fine-grained, four-tiered indicator taxonomy. Concurrently, an open-source indicator repository comprising 162 validated metrics is established. The study identifies critical implementation challenges—including cross-departmental coordination, value attribution, and dynamic adaptability—and provides both a practical classification tool and theoretical foundations to advance data asset management practices and evidence-based decision-making.

Data quality assessment in data monetization remains fragmented and misaligned with value creation. Method: This study proposes an integrative data quality taxonomy grounded in the Balanced Scorecard (BSC), mapping over one hundred generic and domain-specific metrics—via systematic literature review and multidimensional KPI clustering—to the BSC’s four perspectives (financial, customer, internal processes, learning & growth), yielding a hierarchical framework comprising foundational, contextual, resolution, and specialized quality sub-dimensions. Contribution/Results: It innovatively positions data quality as a strategic connector within the BSC, enabling cross-dimensional alignment between technical evaluation and executive decision-making. Empirically grounded and extensible, the framework significantly improves data valuation accuracy, customer trust, operational efficiency, and innovation enablement—advancing data quality management toward sustainable value creation.

To address challenges in cloud-native communication/networking services—including ambiguous SLI/SLO definitions, high expertise barriers for specialized monitoring, and low cross-organizational trust in metrics—this paper proposes the first SRE platform integrating generative AI, federated learning, and blockchain. Methodologically, it introduces federated learning for collaborative, privacy-preserving SLI metric discovery across distributed environments; employs QLoRA-finetuned Llama-3-8B to enable intelligent, context-aware SLI/SLO generation; and leverages smart contracts and NFTs on-chain to immutably attest and audit metrics. The platform is compatible with Prometheus/Mimir, supports lightweight deployment, and was validated on Open5GS 5G core network, demonstrating effective automated SLO management. It simultaneously ensures data privacy, system transparency, and engineering practicality.

This work addresses the unreliability of developer productivity dashboards, which often stems from ad hoc scripts that introduce undetected silent data gaps, eroding organizational trust. To resolve this, we propose a robust ELT pipeline grounded in DAG-based orchestration and the Medallion architecture, decoupling data extraction from transformation to preserve the immutability of raw data. Our approach introduces a state-driven dependency scheduling mechanism and, for the first time, treats metric pipelines as production-grade distributed systems. We emphasize the critical role of immutable raw history in enabling reliable metric redefinition. This methodology significantly enhances data reliability and freshness while effectively eliminating silent failures, thereby restoring organizational confidence in DevOps metrics.

This study addresses the challenges faced by SAE Level 4 autonomous driving systems in handling internal and external disturbances and faults, which are often exacerbated by a lack of stakeholder consensus on performance metrics and interface requirements, leading to non-traceable architectural decisions and inefficient communication. To overcome these issues, this work proposes a process-oriented engineering methodology that employs structured steps to harmonize multi-stakeholder requirements, explicitly define the performance metrics and interface specifications necessary for self-awareness and self-adaptation capabilities, and systematically integrate traceability and knowledge transfer mechanisms into the architecture design process. Validated within the autotech.agil project, the approach significantly enhances requirement consistency, decision transparency, and collaboration efficiency, while yielding key practical insights and directions for future improvement.

This paper addresses the challenge of real-time acquisition and exchange of Key Performance Indicators (KPIs) across multi-vendor equipment in 5G and beyond networks. We propose a KPI extraction and exchange framework compatible with both standardized/commercial components and proprietary tools. Leveraging 3GPP-standard interfaces (e.g., N4, N6, N11), we conduct systematic empirical comparisons of three KPI collection techniques—active probing, passive traffic mirroring, and API polling—across latency, sampling granularity (down to millisecond-level), signaling load sensitivity, and deployment overhead. To our knowledge, this is the first cross-vendor, multi-dimensional empirical evaluation that quantifies performance boundaries and identifies precise applicability conditions for each method. The proposed framework enables on-demand KPI acquisition and protocol-level interoperability, providing telecom operators with reusable, evidence-based guidelines for intelligent network operations and closed-loop optimization.

This study addresses the lack of systematic evaluation of data quality tools with respect to their measurement capabilities and integration with large language models (LLMs). It presents the first multidimensional assessment framework grounded in real-world enterprise use cases, systematically evaluating six prominent tools—including open-source solutions such as Great Expectations and Deequ, as well as commercial platforms like Informatica and Experian—across dimensions including rule definition, duplicate detection, metric aggregation, and uncertainty handling, along with their LLM integration mechanisms. The findings reveal that commercial tools offer more comprehensive functionality and初步 support for LLM-assisted rule generation, whereas open-source tools provide greater flexibility at the cost of higher implementation effort. Notably, none of the evaluated tools currently enable direct LLM-based data validation. This work provides empirical guidance for selecting data quality tools and advancing their integration with LLMs.

To address the challenges of fragmented engineering data and inefficient, inconsistent manual reporting in large-scale software development, this paper proposes and implements a centralized engineering productivity analytics framework supporting near-real-time aggregation and visualization. Methodologically, the framework introduces a dual-mode data storage architecture coupled with a precomputation engine, integrated with scheduled data ingestion (cron), proactive alerting, and role-based access control (RBAC). It unifies heterogeneous data from multiple source systems via a dual-schema design and leverages Metabase to enable cross-dimensional visual analytics—spanning development efficiency, software quality, and operational effectiveness. Empirical evaluation demonstrates that the deployed system reduces average weekly manual reporting effort by 20 person-hours, significantly accelerates bottleneck identification latency, and concurrently improves engineering decision responsiveness and platform scalability.

This study addresses the absence of quantifiable and comparable metrics for evaluating the sustainability of Decentralized Autonomous Organizations (DAOs), which hinders the identification of governance risks and design flaws. To bridge this gap, the work operationalizes a theoretically grounded DAO sustainability KPI framework into deployable software infrastructure, implementing a multi-chain data pipeline and an interactive dashboard. Leveraging on-chain governance and token event data, the system computes a composite sustainability score (0–12) across four dimensions: participation, treasury accumulation, voting efficiency, and decentralization. It supports cross-chain data standardization and transparent client-side scoring. Validated on over 50 active DAOs, 6,930 proposals, and 317,000 voting addresses, the system reveals prevalent governance patterns such as low participation and proposal concentration. The codebase and data architecture are publicly open-sourced.

This work addresses the lack of fine-grained observability in edge-to-cloud continuum systems operating under heterogeneous and dynamic conditions, which hinders adaptive control needed to meet performance objectives. The paper proposes an application-level observability framework that uniquely integrates developer-instrumented telemetry with SLO-aware feedback mechanisms to enable real-time monitoring and autonomous regulation across the edge–cloud continuum. Built upon OpenTelemetry, Prometheus, K3s, and Chaos Mesh, the framework delivers end-to-end observability and elasticity management. Evaluated in a video processing scenario, it significantly enhances system scalability, fault tolerance, and responsiveness, effectively preserving critical quality-of-service metrics such as frame rate, latency, and detection accuracy.

This work addresses the lack of actionable, deployment-ready evaluation mechanisms in current LLM/RAG applications, which hinders effective translation of offline metrics into informed deployment decisions. The authors propose an integrated readiness assessment framework that uniquely combines multidimensional indicators—including workflow success rate, compliance, factual consistency, retrieval hit rate, cost, and latency. By leveraging automated benchmarking, OpenTelemetry-based observability, CI quality gates, and Pareto frontier analysis, the framework generates scenario-weighted readiness scores to guide deployment choices. Empirical validation on FiQA, SciFact, and ticket-routing tasks demonstrates the framework’s ability to reliably distinguish model readiness levels and successfully intercept unsafe prompt variants within CI pipelines, thereby closing the loop from evaluation to deployment.

To address the dual challenges of end-to-end performance monitoring gaps and SLA-aware resource allocation under constrained telemetry budgets in 6G network slicing, this paper models slice monitoring as a closed-loop control problem. It introduces the novel concept of “telemetry primitive contracts” to formally specify minimal data-plane capabilities required for SLA compliance. We further propose an SLA-criticality-driven dynamic resource scheduling mechanism and design a change-triggered In-band Network Telemetry (INT) coordination architecture. Evaluated on programmable switches and large-scale simulations, our approach achieves four times higher monitoring accuracy for critical slices compared to static baselines. The change-triggered INT scheme significantly outperforms existing telemetry primitives while strictly adhering to contract constraints. To the best of our knowledge, this is the first solution enabling SLA-sensitive, end-to-end visible, and resource-adaptive real-time slice monitoring.