RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

2026-07-06

AI Summary

Existing vulnerability analysis benchmarks often rely on non-compilable code snippets, support only binary classification, and overlook data contamination risks, making them inadequate for evaluating large language models’ (LLMs’) ability to detect vulnerabilities in real-world Rust code. This work proposes the first compilable, contamination-aware, multi-granularity benchmark for Rust vulnerabilities, encompassing crate-, file-, and function-level semantically equivalent variants. It supports binary classification, CWE categorization, and line-level localization tasks, and incorporates an adversarial mutation mechanism to assess model robustness. Experiments show that leading LLMs achieve binary classification accuracies of 56–65%, yet attain only around 20% F1 score on line-level localization—a performance that further drops by 27% under adversarial prompting—highlighting their significant limitations in precise vulnerability localization.

Abstract

LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56-65% range, but line localization F1 stays near 20%, and adversarial cues drop line F1 by about 27%.

Problem

• vulnerability analysis
• benchmarking
• data contamination
• compilable code
• Rust

Innovation

• compilable benchmark
• contamination-aware
• vulnerability localization
• code mutation
• Rust security