JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
This work addresses the critical limitations of existing Java vulnerability detection benchmarks, which inadequately balance realistic scenario partitioning, evaluation consistency, and data leakage prevention. To bridge this gap, we construct a high-quality dataset comprising approximately 30,600 methods spanning 1,740 CVEs, introduce a leakage-aware evaluation protocol alongside a pretraining contamination auditing mechanism, and design five practical data splitting strategies. We further establish the first unified evaluation framework that seamlessly integrates encoder-only models, local generative models, and API-based large language models, enabling single-command, multi-backend cross-model assessment for CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder, and major LLM APIs. The project releases all data, code, fine-tuned models, and twelve reference detectors, accompanied by a complete evaluation pipeline demonstration.