Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla

2026-06-16

AI Summary

This work addresses the high false positive rate (12.5%) and false negative rate (6.8%) of Mozilla’s existing T-test–based performance anomaly detection system, which hampers continuous integration efficiency. The authors introduce the first benchmark dataset comprising 174 engineer-annotated performance time series and conduct a systematic evaluation of 25 change-point detection algorithms combined with 15 ensemble strategies. They propose an ensemble voting mechanism that integrates offline, online, and hybrid methods to effectively mitigate the precision–recall trade-off. Experimental results and engineer feedback demonstrate that the proposed approach improves the F1-score by 11% over the original system and has been successfully integrated into Mozilla’s performance engineering infrastructure.

Abstract

Software performance regressions can have significant business consequences, making automated detection a critical component of modern continuous integration pipelines. At Mozilla, performance anomaly detection is handled by Perfherder, Mozilla's performance engineering management system that relies on a Student's T-test-based approach to flag regressions across hundreds of daily code changes. However, our preliminary analysis of one year of Mozilla performance data reveals that 12.5% of generated alert groups are false positives, while approximately 6.8% of them contain regressions missed by the automated system. This paper presents an empirical study evaluating 25 change-point detection (CPD) methods and 15 ensemble approaches as alternatives to Mozilla's current method. We construct a ground-truth dataset of 174 performance time series manually annotated by eleven Mozilla performance engineers, representing one of the first practitioner-annotated CPD benchmarks for performance engineering. Our results show that while offline and hybrid CPD methods improve recall over Mozilla's method, they do so at a high cost to precision. Ensemble voting strategies alleviate this trade-off and offer more consistent performance, resulting in 11% improvement in the F1-score. We validate the experimental results through a practitioner survey and report on lessons learned from integrating the best methods into Mozilla's performance engineering system.

Problem

• performance anomaly detection
• change point detection
• false positives
• missed regressions
• continuous integration

Innovation

• change-point detection
• performance anomaly detection
• ensemble methods
• ground-truth benchmark
• continuous integration