The Case for Multi-Version Experimental Evaluation (MVEE)

2026-03-29

AI Summary

This work addresses a critical yet often overlooked source of evaluation bias in database research: performance variability arising from non-deterministic compiler behavior across different builds, even when the source code remains unchanged. To systematically identify and mitigate this threat to reproducibility, the paper introduces the Multi-Version Experimental Evaluation (MVEE) framework. MVEE automatically detects “build anomalies” through assembly-level analysis, materializes them as distinct method versions, and incorporates these variants into a unified evaluation protocol. By explicitly accounting for implicit compilation-induced differences, MVEE substantially enhances the reliability and expressiveness of experimental results, ensuring that performance comparisons reflect genuine algorithmic characteristics rather than artifacts of the build process. This approach strengthens the scientific rigor and credibility of empirical studies in database systems research.

Technology Category

• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

In the database community, we typically evaluate new methods based on experimental results, which we produce by integrating the proposed method along with a set of baselines in a single benchmarking codebase and measuring the individual runtimes. If we are unhappy with the performance of our method, we gradually improve it while repeatedly comparing to the baselines, until we outperform them. While this seems like a reasonable approach, it makes one delicate assumption: We assume that across the optimization workflow, there exists only a single compiled version of each baseline to compare to. However, we learned the hard way that in practice, even though the source code remains untouched, general purpose compilers might still generate highly different compiled code across builds, caused by seemingly unrelated changes in other parts of the codebase, leading to flawed comparisons and evaluations. To tackle this problem, we propose the concept of Multi-Version Experimental Evaluation (MVEE). MVEE automatically and transparently analyzes subsequent builds on the assembly code level for occurring "build anomalies" and materializes them as new versions of the methods. As a consequence, all observed versions of the respective methods can be included in the experimental evaluation, highly increasing its quality and overall expressiveness.

Problem

• experimental evaluation
• build anomalies
• compiler variability
• performance comparison
• baseline consistency

Innovation

• Multi-Version Experimental Evaluation
• build anomalies
• compiler variability
• experimental evaluation
• assembly-level analysis