Just-in-Time Catching Test Generation at Meta

2026-01-30

AI Summary

This work addresses the inefficiency of pre-merge defect detection and the high burden of false positives in large-scale backend systems by introducing an "immediate-capture testing" paradigm. The approach dynamically generates high-precision expected-failure tests informed by code change context and features a novel evaluation mechanism that synergistically combines a rule engine with a large language model (LLM) to efficiently prioritize high-value test candidates. Experimental results demonstrate that the proposed method achieves a 4× improvement in defect detection efficiency over traditional fortification testing and a 20× gain compared to flaky-test-based approaches, while reducing manual review effort by 70%. Furthermore, it successfully identified eight real-world defects, four of which could trigger severe production incidents.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production.

Problem

• Just-in-Time testing
• catching tests
• false positives
• bug prevention
• large-scale systems

Innovation

• Just-in-Time Testing
• Catching Tests
• Code-Change-Aware
• False Positive Reduction
• LLM-Based Assessment