VulGuard: An Unified Tool for Evaluating Just-In-Time Vulnerability Prediction Models

2025-07-22

AI Summary

To address critical challenges in just-in-time vulnerability prediction (JIT-VP)—including poor reproducibility, labor-intensive data preprocessing, and inconsistent model evaluation—this paper introduces JIT-VP Toolkit, an open-source, automated framework. It unifies commit history mining, fine-grained code change feature extraction (spanning syntactic and semantic levels), multi-model integration (encompassing traditional machine learning and pre-trained models), and standardized evaluation protocols, with native support for CI/CD integration and benchmarking. The toolkit automates GitHub repository data collection, cleaning, labeling, and feature engineering, substantially lowering experimental barriers. Empirical validation on the FFmpeg and Linux kernel datasets demonstrates that JIT-VP Toolkit improves model reproducibility by over 50% and enhances cross-study comparability. To our knowledge, it is the first end-to-end, scalable, and reproducible infrastructure specifically designed for JIT-VP research.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We present VulGuard, an automated tool designed to streamline the extraction, processing, and analysis of commits from GitHub repositories for Just-In-Time vulnerability prediction (JIT-VP) research. VulGuard automatically mines commit histories, extracts fine-grained code changes, commit messages, and software engineering metrics, and formats them for downstream analysis. In addition, it integrates several state-of-the-art vulnerability prediction models, allowing researchers to train, evaluate, and compare models with minimal setup. By supporting both repository-scale mining and model-level experimentation within a unified framework, VulGuard addresses key challenges in reproducibility and scalability in software security research. VulGuard can also be easily integrated into the CI/CD pipeline. We demonstrate the effectiveness of the tool in two influential open-source projects, FFmpeg and the Linux kernel, highlighting its potential to accelerate real-world JIT-VP research and promote standardized benchmarking. A demo video is available at: https://youtu.be/j96096-pxbs

Problem

• Automates extraction and analysis of GitHub commits for vulnerability prediction
• Integrates state-of-the-art models for training and comparison in JIT-VP research
• Addresses reproducibility and scalability challenges in software security research

Innovation

• Automates GitHub commit extraction and processing
• Integrates state-of-the-art vulnerability prediction models
• Supports repository-scale mining and model experimentation