A Generic and Efficient Python Runtime Verification System and its Large-scale Evaluation

2025-09-08

AI Summary

Existing Python runtime verification (RV) systems suffer from poor generality, limited logical expressiveness, or suboptimal performance. This paper introduces PyMOP—a general-purpose, efficient, and extensible Python RV framework supporting five specification logics, five monitoring algorithms, and three instrumentation strategies. Its key innovations include: (i) the first empirical demonstration that mainstream Java RV algorithms are suboptimal for Python’s execution model, coupled with the design of Python-tailored optimizations; (ii) integration of API specifications for 73 widely used Python libraries; and (iii) rigorous evaluation via dynamic analysis and large-scale unit testing across 1,463 GitHub projects and over 290,000 test cases—achieving up to 1,168.3× speedup over state-of-the-art tools and enabling the detection and repair of 44 real-world defects.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Runtime verification (RV) now scales for testing thousands of open-source Java projects, helping find hundreds of bugs. The popular Python ecosystem could use such benefits. But, today's Python RV systems are limited to a domain or specification logic, or slow. We propose PyMOP, a generic, extensible, and efficient RV system for Python. PyMOP supports five logics, implements five existing monitoring algorithms, ships with 73 API specs of Python and widely-used libraries, supports three instrumentation strategies, and users can easily add more of these. On 290,133 unit tests in 1,463 GitHub projects, we find mainly that (i) the default monitoring algorithm for Java is often not the fastest for Python; (ii) PyMOP is up to 1,168.3x faster than two recent dynamic analysis systems; and (iii) 44 of 121 bugs that PyMOP helped find so far were fixed by developers. PyMOP's generality and efficiency position it well as an excellent platform for the next advances on RV for Python.

Problem

• Developing a generic runtime verification system for Python
• Overcoming limitations of current slow and domain-specific Python RV tools
• Enabling efficient large-scale bug detection in Python projects

Innovation

• Generic extensible efficient Python runtime verification system
• Supports multiple logics monitoring algorithms instrumentation strategies
• Large-scale evaluation shows significant speedup and bug detection