JS-TOD: Detecting Order-Dependent Flaky Tests in Jest

2025-08-30

AI Summary

This paper addresses flaky tests in Jest caused by ordering dependencies—primarily due to shared files or mock state across test cases, leading to non-deterministic outcomes. To systematically detect such dependencies, we propose a dynamic test reordering framework operating at three granularities: individual test cases, `describe` blocks, and test suites. Our method integrates test extraction, multi-round randomized re-execution, and configurable retries (defaulting to 10 rounds) to reliably expose ordering-sensitive defects. As the first fine-grained, configurable test randomization framework tailored for Jest, our approach empirically identifies shared mock state as the predominant root cause of flakiness. Evaluated on real-world projects, it successfully detects diverse ordering-dependent defects—including state leakage and mock contamination—demonstrating both technical efficacy and engineering practicality.

Technology Category

• Planning, Routing, and Scheduling: Deterministic Planning
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We present JS-TOD (JavaScript Test Order-dependency Detector), a tool that can extract, reorder, and rerun Jest tests to reveal possible order-dependent test flakiness. Test order dependency is one of the leading causes of test flakiness. Ideally, each test should operate in isolation and yield consistent results no matter the sequence in which tests are run. However, in practice, test outcomes can vary depending on their execution order. JS-TOD employed a systematic approach to randomising tests, test suites, and describe blocks. The tool is highly customisable, as one can set the number of orders and reruns required (the default setting is 10 reorder and 10 reruns for each test and test suite). Our evaluation using JS-TOD reveals two main causes of test order dependency flakiness: shared files and shared mocking state between tests.

Problem

• Detecting order-dependent flaky tests in Jest
• Identifying causes of test execution variability
• Automating test reordering and rerunning for reliability

Innovation

• Randomizing test execution order systematically
• Customizable reordering and rerun configurations
• Detecting shared files and mocking state issues