If-T: A Benchmark for Type Narrowing

2025-08-05

AI Summary

Runtime type tests in dynamic languages necessitate type narrowing, yet existing narrowing mechanisms lack standardized evaluation criteria, suffer from formal complexity, and exhibit unclear practical utility. Method: We introduce If-T—the first language-agnostic benchmark for type narrowing design—comprising minimal, controlled positive and negative test cases that systematically characterize the verification capabilities of gradual type systems across core scenarios, including logical reasoning and user-defined predicates. If-T explicitly defines precision–performance trade-off principles to enable fair cross-language comparison. Contribution/Results: Built via literature analysis, documentation review of mainstream languages (TypeScript, Flow, Typed Racket, mypy, Pyright), and empirical evaluation, If-T reveals critical disparities among systems in flow-sensitive reasoning and custom assertion handling. All artifacts are open-sourced, providing a reproducible, extensible evaluation framework for type system design and implementation.

Technology Category

• Knowledge Representation and Reasoning: Argumentation
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

**Context:** The design of static type systems that can validate dynamically-typed programs (**gradually**) is an ongoing challenge. A key difficulty is that dynamic code rarely follows datatype-driven design. Programs instead use runtime tests to narrow down the proper usage of incoming data. Type systems for dynamic languages thus need a **type narrowing** mechanism that refines the type environment along individual control paths based on dominating tests, a form of flow-sensitive typing. In order to express refinements, the type system must have some notion of sets and subsets. Since set-theoretic types are computationally and ergonomically complex, the need for type narrowing raises design questions about how to balance precision and performance. **Inquiry:** To date, the design of type narrowing systems has been driven by intuition, past experience, and examples from users in various language communities. There is no standard that captures desirable and undesirable behaviors. Prior formalizations of narrowing are also significantly more complex than a standard type system, and it is unclear how the extra complexity pays off in terms of concrete examples. This paper addresses the problems through If-T, a language-agnostic **design benchmark** for type narrowing that characterizes the abilities of implementations using simple programs that draw attention to fundamental questions. Unlike a traditional performance-focused benchmark, If-T measures a narrowing system's ability to validate correct code and reject incorrect code. Unlike a test suite, systems are not required to fully conform to If-T. Deviations are acceptable provided they are justified by well-reasoned design considerations, such as compile-time performance. **Approach:** If-T is guided by the literature on type narrowing, the documentation of gradual languages such as TypeScript, and experiments with typechecker implementations. We have identified a set of core technical dimensions for type narrowing. For each dimension, the benchmark contains a set of topics and (at least) two characterizing programs per topic: one that should typecheck and one that should not typecheck. **Knowledge:** If-T provides a baseline to measure type narrowing systems. For researchers, it provides criteria to categorize future designs via its collection of positive and negative examples. For language designers, the benchmark demonstrates the payoff of typechecker complexity in terms of concrete examples. Designers can use the examples to decide whether supporting a particular example is worthwhile. Both the benchmark and its implementations are freely available online. **Grounding:** We have implemented the benchmark for five typecheckers: TypeScript, Flow, Typed Racket, mypy, and Pyright. The results highlight important differences, such as the ability to track logical implications among program variables and typechecking for user-defined narrowing predicates. **Importance:** Type narrowing is essential for gradual type systems, but the tradeoffs between systems with different complexity have been unclear. If-T clarifies these tradeoffs by illustrating the benefits and limitations of each level of complexity. With If-T as a way to assess implementations in a fair, cross-language manner, future type system designs can strive for a better balance among precision, annotation burden, and performance.

Problem

• Designing type systems to validate dynamically-typed programs gradually
• Balancing precision and performance in type narrowing mechanisms
• Establishing a benchmark to assess type narrowing system capabilities

Innovation

• Language-agnostic benchmark for type narrowing
• Core technical dimensions with example programs
• Balances precision, performance, and annotation burden