Exceptional Behaviors: How Frequently Are They Tested?
This study addresses the lack of systematic understanding regarding test coverage of anomalous behaviors in real-world systems, particularly anomalies that do not propagate to test failures. For the first time, it jointly examines both propagated and non-propagated exceptions by dynamically instrumenting 25 Python projects, monitoring 5,372 methods, 17.9 million method calls, and 1.4 million exceptions. The analysis reveals that 21.4% of methods raise exceptions, with approximately 20% of these doing so frequently—exhibiting a median rate of one exception per ten invocations. These findings challenge the conventional assumption that exceptions are rare events and demonstrate that anomalous behavior is far more prevalent in practice than previously believed.