CornerCase: Automated Extremal Testing of Protocol Implementations using LLMs

2026-06-27

AI Summary

This work addresses the longstanding challenge of detecting boundary vulnerabilities in network protocol implementations, which are often missed by traditional fuzzing and model-driven approaches due to their inability to adequately cover edge cases specified in protocol standards. The authors propose an automated extreme-value testing method powered by large language models (LLMs), which systematically parses protocol specifications to extract explicit validity constraints section by section and generates targeted test cases near constraint boundaries. By integrating differential testing, the approach uncovers inconsistent behaviors across multiple protocol implementations. This is the first study to leverage LLMs for both boundary constraint extraction and extreme-value test case generation, leading to the discovery of 42 previously unknown anomalies across HTTP, DNS, BGP, SMTP, and QUIC protocols—26 of which were confirmed as vulnerabilities, with 18 already patched—demonstrating a significant advance in boundary vulnerability detection.

Abstract

Many software bugs in network protocol implementations arise near specification boundaries, such as inputs just within or outside allowed ranges, or messages that are valid in isolation but invalid in a given state. From the SSL Heartbleed exploit to TCP Christmas Tree packets, boundary inputs have repeatedly exposed critical weaknesses, yet remain under-tested by existing techniques such as fuzzing and model-based testing. We present CornerCase, an automated extremal testing approach that systematically targets such boundary behaviors. Our key idea is to decompose test generation into two stages: first, large language models (LLMs) extract explicit validity constraints from protocol specifications (e.g., RFCs) in a structured, section-by-section manner; second, extremal test cases are generated at or near the boundary of each constraint. These tests are executed across multiple implementations, and differential testing identifies inconsistencies. We evaluate CornerCase on widely used implementations of HTTP, DNS, BGP, SMTP, and QUIC, uncovering many previously unknown bugs. For example, the HTTP server h2o enters a redirect loop when processing URLs containing encoded null bytes. Overall, we used CornerCase to identify and file 42 anomalies; to date 26 have been acknowledged as bugs and 18 fixed, with others under active investigation

Problem

• boundary testing
• protocol implementation
• software bugs
• extremal inputs
• specification compliance

Innovation

• extremal testing
• large language models
• protocol specification
• boundary analysis
• differential testing