data-driven decision making

In data-driven economies, organizations lack systematic frameworks for evaluating and managing data value within internal business processes. To address this gap, this study develops a comprehensive data value assessment framework grounded in the Balanced Scorecard’s internal process perspective, integrating three interrelated dimensions: data quality, governance compliance, and operational efficiency. It introduces a novel, multi-layered taxonomy of data value—spanning technological, organizational, and regulatory dependencies—that resolves metric redundancy and establishes cross-dimensional conceptual linkages. Through systematic literature review, theoretical modeling, indicator clustering, and taxonomy design, the research produces a scalable, reusable data value metrics system. This system underpins standardized data valuation models and decision-support systems, offering both a methodological foundation and actionable implementation pathways for cross-sectoral data assetization. (149 words)

In large-scale, low-signal A/B testing at technology companies, existing decision rules lack unbiased and comparable evaluation criteria, leading to severe bias in estimating cumulative North Star metric gains under low signal-to-noise ratios. To address this, we propose the first cross-validation–based unbiased estimator for cumulative treatment effects, integrating causal inference principles with meta-analytic thinking to overcome the high bias inherent in conventional plug-in estimators in weak experiments. We further develop a quantitative evaluation framework for decision rules centered on cumulative North Star metric gains. Evaluated on 123 historical experiments at Netflix, our rule increases cumulative gains by 33% and has been deployed in production. Our core contributions are: (i) the first unbiased estimator for cumulative treatment effects in online experimentation, and (ii) a novel, industrially grounded paradigm for benchmarking and comparing decision rules on a common, interpretable metric.

Existing A/B testing suffers from long cycles and high operational costs, hindering rapid evaluation of product decisions on user retention and long-term profitability; offline methods, in contrast, exhibit low reliability and weak causal identification. This paper proposes a lightweight offline scenario analysis framework that integrates user behavioral log modeling, counterfactual inference, causal chain estimation of business metrics, and scalable simulation—enabling hypothesis validation within minutes. The framework introduces three novel capabilities: (1) automated generation of hypotheses grounded in large-scale business metric correlations; (2) multi-objective trade-off assessment under shifts in consumption structure; and (3) trend forecasting of long-term metrics such as retention rate and user lifetime value. Empirical evaluation on real-world production data demonstrates significantly higher prediction accuracy than baseline approaches, substantially improving both the efficiency and scientific rigor of product decision-making.

In A/B testing, control variates and regression adjustment are widely used variance reduction techniques, yet their theoretical relationship remains unclear, their methodological frameworks are disjointed, and both have long been confined to design-driven paradigms. Method: This paper establishes, for the first time, a formal equivalence between these two approaches and proposes a novel grouped coefficient estimation method that unifies design-based and model-based estimation frameworks—enabling a paradigm shift from design-driven to model-driven inference. Contribution/Results: Theoretical analysis demonstrates improved estimation accuracy and statistical power. Empirical validation on millions of real-world experiments at ByteDance confirms efficacy: the proposed method has been fully deployed in its online experimentation platform, yielding an average 12.3% increase in statistical significance and a 19.6% improvement in detection sensitivity.

Business professionals—non-technical domain experts—lack appropriate tools and methodologies for effective what-if analysis (WIA), hindering data-informed decision-making. Method: We conducted a two-phase mixed-methods user study—comprising contextual interviews and in-situ task-based evaluations—to systematically characterize their analytical behaviors for the first time. Contribution/Results: Based on empirical findings, we propose three domain-grounded design principles: business-contextual data preparation, risk-aware assessment, and domain-knowledge integration. We implemented and validated these principles in an interactive visual analytics prototype. The study identifies three critical support gaps, empirically confirms that six classes of what-if techniques significantly improve decision efficiency and confidence, and yields eight actionable design guidelines for commercial business intelligence systems. This work bridges a key theoretical and practical gap in WIA research concerning non-technical users.

Accurately estimating the causal effects of long-term product interventions—such as UI redesigns or recommendation algorithm updates—in digital platforms remains challenging, as conventional short-term A/B tests fail to capture delayed and evolving impacts. To address this, we propose the first causal inference framework specifically designed for estimating long-term treatment effects. Our approach disentangles time-varying confounding from lagged treatment effects by explicitly modeling treatment duration as a key covariate. It integrates structural time-series modeling, doubly robust estimation, and dynamic causal graphs to enable counterfactual effect estimation without requiring costly long-duration experiments. Evaluated on real-world platform data, our method reduces long-term effect estimation error by 42% and achieves high-fidelity predictions across core metrics—including user retention rate and click-through rate—thereby significantly improving both the reliability and efficiency of long-horizon strategy evaluation.

Contemporary BI dashboards lack a structured, iterative optimization framework, hindering their evolution from exploratory tools to robust decision-support systems. Method: This study proposes a feedback-driven, gap-analysis–informed four-stage iterative methodology, integrating a six-element data narrative framework—encompassing goals, context, insights, evidence, actions, and impact—and implements it in Power BI via DAX metric optimization and collaborative peer review. Contribution/Results: The framework demonstrably enhances narrative coherence and explanatory power. Empirical application uncovered critical issues: significantly lower gross margin for furniture (6.94% vs. 13.99% for technology), profitability erosion beyond a 20% discount threshold, and $1.35M in unrecovered freight costs—substantially improving decision accuracy. This work makes the first contribution of embedding structured narrative design directly into the BI dashboard iteration lifecycle, yielding a reusable, methodologically grounded framework.

In A/B testing, rigorously evaluating novel estimation algorithms—when the true treatment effect is unobserved—remains a fundamental methodological challenge. This paper establishes, for the first time, a comprehensive theoretical framework for estimation and inference based on sample splitting: it derives the asymptotic distribution of sample-split estimators and characterizes their bias structure relative to full-sample performance; introduces a bias–variance trade-off analytical paradigm and proposes a correction-based confidence interval construction method. Leveraging statistical inference, asymptotic theory, Monte Carlo simulation, and empirical validation, the framework enables robust, production-grade evaluation of new algorithms within industrial A/B testing platforms. Theoretical results are thoroughly validated via simulation studies. The proposed infrastructure enhances A/B testing methodology by delivering an interpretable, reproducible, and deployable evaluation system.

This work addresses the inefficiencies in clinical trial design and analysis that hinder drug development success rates and timelines. It proposes establishing a dedicated statistical methodology team within pharmaceutical companies as a strategic investment, embedded through a systematic organizational structure and cross-functional collaboration mechanisms—both internally across departments and externally with academic and regulatory partners—to break down information silos. By integrating advanced statistical modeling, optimized clinical trial designs, and other high-impact quantitative methodologies, this team significantly enhances R&D efficiency, shortens development cycles, and strengthens the scientific rigor and likelihood of success in clinical decision-making.

This paper addresses the critical problem of variance reduction via stratified sampling in online A/B testing. We propose an efficient stratification variable subset selection algorithm that dynamically evaluates the marginal contribution of each variable to estimation variance through layer-wise simulation of the stratification process, enabling precise identification of high-information stratification variables—even under multivariate correlation. Unlike conventional approaches relying on pairwise correlation or heuristic filtering, our method directly optimizes for variance minimization, ensuring both theoretical interpretability and computational efficiency. Experiments on synthetic and real-world business datasets demonstrate that our approach reduces estimation variance by 18%–32% on average compared to classical methods such as covariate adjustment and CUPED. This translates into significantly improved statistical power and experimental sensitivity, facilitating faster and more reliable causal inference in production A/B testing environments.

Current what-if analysis lacks a unified conceptual framework, leading to terminological inconsistency across domains, structural ambiguity, and divergent interpretations. To address this, we conduct a systematic review of 141 papers in visual analytics and human-computer interaction, proposing Praxa—the first integrative framework that unifies scenario modeling, sensitivity analysis, and counterfactual analysis under a coherent paradigm. Praxa formally defines the underlying motivations, core components (hypothesis generation, intervention modeling, outcome evaluation), and a taxonomy of analytical types. It establishes a standardized terminology and structured model, exposing critical challenges including interpretability, causal modeling fidelity, and alignment with user intent. By clarifying conceptual boundaries and operational relationships among methods, Praxa significantly enhances cross-domain conceptual consistency and application clarity. The framework provides a rigorous foundation for theoretical advancement and the design of next-generation interactive analytical tools.