The Polyglot's Dilemma: Conformance Testing a Dozen Specs in as Many Languages

📅 2026-08-18
📈 Citations: 0
Influential: 0
📄 PDF
🤖 AI Summary
为确保MongoDB在多种编程语言中的客户端库行为一致,开发了基于YAML的统一测试格式,减少了非一致性错误。
📝 Abstract
MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, we developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. We describe the evolution from many ad-hoc formats to a Unified Test Format, which allowed us to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests (though results varied). We report lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.
Problem

Research questions and friction points this paper is trying to address.

conformance testing
specification-based testing
consistent behavior
multiple programming languages
client libraries
Innovation

Methods, ideas, or system contributions that make the work stand out.

specification-based testing
Unified Test Format
YAML
conformance testing
cross-language consistency
🔎 Similar Papers
No similar papers found.
A
A. Jesse Jiryu Davis
MongoDB Research, New York, New York, USA
J
Jeremy Mikola
Independent, Hoboken, New Jersey, USA
J
Jeff Yemin
MongoDB, Inc., New York, New York, USA