The Fifth International Verification of Neural Networks Competition (VNN-COMP 2024): Summary and Results

2024-12-28

AI Summary

Neural network formal verification tools lack unified, reproducible evaluation standards. Method: This work initiated and organized the 2024 Verification of Neural Networks Competition (VNN-COMP’24), introducing a fully automated end-to-end evaluation framework. It leverages standardized ONNX models and the VNN-LIB specification language, executing all submissions on a uniform AWS cloud infrastructure with preconfigured parameters, isolated test sets, and automatic result validation to prevent data leakage. Contribution/Results: VNN-LIB was deployed at an industrial scale for the first time, integrating mainstream techniques including constraint solving and abstract interpretation. Eight international teams participated, completing 12 standard and 8 extended benchmark tasks—covering critical verification scenarios such as robustness and safety. The competition established an authoritative, reproducible performance baseline, significantly advancing verification methodology and standardization.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

This report summarizes the 5th International Verification of Neural Networks Competition (VNN-COMP 2024), held as a part of the 7th International Symposium on AI Verification (SAIV), that was collocated with the 36th International Conference on Computer-Aided Verification (CAV). VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools, encourage the standardization of tool interfaces, and bring together the neural network verification community. To this end, standardized formats for networks (ONNX) and specification (VNN-LIB) were defined, tools were evaluated on equal-cost hardware (using an automatic evaluation pipeline based on AWS instances), and tool parameters were chosen by the participants before the final test sets were made public. In the 2024 iteration, 8 teams participated on a diverse set of 12 regular and 8 extended benchmarks. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.

Problem

• Neural Network Verification
• Performance Evaluation
• Comparative Analysis

Innovation

• ONNX
• VNN-LIB
• AWS Cloud Computing