experimentation

This study addresses the challenge of biased effect estimation in online controlled experiments caused by overlapping tests on shared traffic, which hinders accurate assessment of feature interactions. To resolve this, the authors propose Multi-Experiment Analysis (MEA), a method grounded in statistical modeling and causal inference that consistently estimates joint effects under arbitrary partial or full overlap and multi-variant settings—without requiring predefined factorial designs or constrained traffic allocation. MEA uniquely enables, without coordination overhead, the simultaneous modeling of bias-corrected individual effects, joint effects for any combination of variants, and conditional effects. Simulations confirm the estimator’s consistency and nominal confidence interval coverage, and the approach has been successfully deployed in large-scale production systems across multiple real-world business applications.

Reproducibility, scalability, and cross-team collaboration in quantum computing experiments remain significant challenges. To address these, this work introduces a structured experimental tracking workflow—marking the first systematic adoption of the MLflow framework in quantum research. It enables unified, standardized logging of parameters, metrics, source code, and quantum circuits within hybrid classical-quantum experiments. The approach seamlessly integrates mainstream quantum programming frameworks—including Qiskit and PennyLane—and supports versioned experiment records and exact result reproducibility. Empirical evaluation demonstrates that the proposed methodology substantially improves reproducibility and collaborative efficiency in quantum software development, reduces knowledge retention overhead, and establishes a scalable engineering infrastructure for large-scale quantum algorithm development and interdisciplinary collaboration.

Scientific machine learning experiments often suffer from distorted performance evaluations due to poor experimental design and inconsistent documentation. To address this, we propose a principled framework for ML experimentation tailored to scientific research, encompassing data preprocessing, model selection, cross-validation, and reporting—emphasizing reproducibility, fair comparison, and transparency. Our key contributions include two novel quantitative metrics: the Logarithmic Overfitting Ratio (LOR) and Composite Overfitting Score (COS), which jointly characterize overfitting severity and instability across cross-validation folds. Complementing these, we introduce standardized preprocessing protocols, rigorously defined strong baselines, and modular visualization templates for diagnostic analysis. Empirical evaluation demonstrates that our framework substantially enhances experimental rigor, reproducibility, and result credibility in scientific ML. It further enables robust performance assessment and cross-study comparability, providing systematic support for establishing reliable benchmarks.

In A/B testing, control variates and regression adjustment are widely used variance reduction techniques, yet their theoretical relationship remains unclear, their methodological frameworks are disjointed, and both have long been confined to design-driven paradigms. Method: This paper establishes, for the first time, a formal equivalence between these two approaches and proposes a novel grouped coefficient estimation method that unifies design-based and model-based estimation frameworks—enabling a paradigm shift from design-driven to model-driven inference. Contribution/Results: Theoretical analysis demonstrates improved estimation accuracy and statistical power. Empirical validation on millions of real-world experiments at ByteDance confirms efficacy: the proposed method has been fully deployed in its online experimentation platform, yielding an average 12.3% increase in statistical significance and a 19.6% improvement in detection sensitivity.

Existing behavioral malware detection research heavily relies on sandbox-derived features, leading to severe performance degradation—accuracy drops to 20%–50%, far below the reported >90%—when deployed on real endpoints. This exposes three fundamental challenges: label noise, distribution shift, and spurious feature reliance. To address this, we conduct the first large-scale empirical evaluation on real endpoints and propose a robust end-to-end training framework tailored for endpoint environments. Our approach integrates behavioral trajectory modeling, cross-environment distribution alignment, noise-robust learning, and telemetry-driven training leveraging real-world endpoint telemetry. Experiments demonstrate a 5%–30% relative improvement in detection accuracy over sandbox-trained baselines. Crucially, our work shifts the detection paradigm from sandbox-centric training to direct training on endpoint data. As part of this contribution, we release the first publicly available benchmark dataset comprising authentic endpoint behavioral trajectories, enabling reproducible, realistic evaluation of behavioral malware detection systems.

This study addresses the widespread lack of systematic training in instrumentation software and machine learning tools among early-career researchers in high-energy physics, a gap that significantly hinders their research efficiency and professional development. Focusing specifically on this cohort’s practical needs and deficiencies regarding open-source software and machine learning education, the project collected feedback from 174 early-career researchers through a structured survey and employed statistical analysis to evaluate the accessibility and quality of existing training programs. The findings reveal that approximately 70% of respondents have received no such training. These results provide empirical evidence to inform the design of targeted, effective training frameworks aimed at enhancing the computational and analytical competencies of young scientists in the field.

This study addresses the risk of inference bias and potential failure of the CUPED method in online A/B testing under complex experimental conditions. The authors systematically investigate five critical issues related to variance reduction with CUPED, and for the first time delineate its applicability boundaries in designs such as multi-arm experiments and two-stage sampling. To overcome these limitations, they propose a robust variance estimation approach tailored to such settings. Through rigorous theoretical analysis and large-scale empirical validation, the proposed method significantly improves inference accuracy. The solution has been successfully deployed in ByteDance’s experimentation platform, demonstrating its practical effectiveness and scalability.

This work addresses the inefficiency and steep learning curve researchers often encounter when trying to map academic papers to their corresponding implementation code. To bridge this gap, the authors propose an automated tool powered by large language models (LLMs) that achieves cross-modal semantic alignment between scholarly texts and source code for the first time. By integrating program analysis techniques, the method automatically identifies code segments that implement specific research ideas described in a paper and generates high-quality traceability mappings. This approach substantially reduces the manual effort required for alignment, enhances the comprehensibility of research software, and improves reproducibility. Preliminary experiments demonstrate the tool’s practicality and effectiveness in real-world scenarios.

This study addresses the frequent conflict between surrogate metrics and core North Star metrics in online experimentation, where a principled framework for their integrated decision-making has been lacking. The authors propose a dynamic optimal fusion method that establishes, for the first time, a formal trade-off framework unifying metric quality and experimental power within a single decision system: it prioritizes the North Star metric when experimental power increases, while upweighting the surrogate metric when its validity is high. The approach estimates optimal fusion weights from historical experiment data by jointly leveraging statistical power analysis and surrogate metric validity assessment. Deployed on Netflix’s experimentation platform, this method significantly enhances decision efficiency and the precision of resource allocation in online experiments.

Environmental drift renders a substantial number of machine learning notebooks irreproducible, hindering scientific reuse and progress. This work proposes MLEModernizer, a novel framework that leverages large language model (LLM)-driven agents to modernize notebooks without relying on environment rollback. Treating the current execution environment as a fixed constraint, the framework iteratively applies three strategies—error repair, runtime optimization, and score calibration—guided by execution feedback. Evaluated on 7,402 irreproducible notebooks, MLEModernizer successfully restores reproducibility in 5,492 cases (74.2%), significantly enhancing the sustainable usability of machine learning engineering assets.