Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript

2025-01-22

AI Summary

Non-deterministic test failures in JavaScript Jest suites—caused by execution-order dependencies—severely undermine test reliability. This paper proposes a systematic detection methodology: applying ten distinct random test-ordering strategies across 100 repeated executions to identify order-dependent flaky tests in 81 GitHub projects, revealing 55 such fragile tests. Crucially, we uncover “shared mocking state” as the predominant root cause in the JavaScript ecosystem—distinct from conventional file- or global-state sharing. Leveraging Jest’s native multi-level (test/suite/describe) randomization, statistical analysis of outcome discrepancies, and manual root-cause investigation, we confirm that mocking state pollution accounts for 42 (76.4%) of the identified fragile tests. This finding highlights a previously underappreciated risk in JavaScript testing: improper mock lifecycle management introduces subtle, non-local interference between tests. Our work provides both theoretical insight into JS testing fragility and actionable guidance—such as stricter mock isolation and deterministic reset policies—for improving frontend test stability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously

Problem

• JavaScript
• Jest
• Test Instability

Innovation

• JavaScript Testing
• Test Order Effect
• Shared Mock State