From Description to Score: Can LLMs Quantify Vulnerabilities?

2025-12-07

AI Summary

This study addresses the subjectivity and time-intensive nature of manual CVSS scoring by systematically evaluating large language models (LLMs)—including ChatGPT, Llama, Grok, DeepSeek, and Gemini—on automated CVSS vector scoring across 31,000+ CVE entries. We propose a multi-model ensemble-based meta-classifier framework and conduct an in-depth error analysis. Our findings reveal, for the first time, that contextual incompleteness and linguistic ambiguity in CVE descriptions constitute the primary bottlenecks limiting LLM scoring accuracy. Experimental results show that LLMs significantly outperform traditional baselines on metrics such as “Impact to Availability,” with ChatGPT-5 achieving the highest precision. However, ensemble strategies yield only marginal gains, and overall performance remains highly dependent on the quality of original vulnerability descriptions. This work provides empirical evidence and actionable insights for improving the reliability of automated vulnerability assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Manual vulnerability scoring, such as assigning Common Vulnerability Scoring System (CVSS) scores, is a resource-intensive process that is often influenced by subjective interpretation. This study investigates the potential of general-purpose large language models (LLMs), namely ChatGPT, Llama, Grok, DeepSeek, and Gemini, to automate this process by analyzing over 31{,}000 recent Common Vulnerabilities and Exposures (CVE) entries. The results show that LLMs substantially outperform the baseline on certain metrics (e.g., extit{Availability Impact}), while offering more modest gains on others (e.g., extit{Attack Complexity}). Moreover, model performance varies across both LLM families and individual CVSS metrics, with ChatGPT-5 attaining the highest precision. Our analysis reveals that LLMs tend to misclassify many of the same CVEs, and ensemble-based meta-classifiers only marginally improve performance. Further examination shows that CVE descriptions often lack critical context or contain ambiguous phrasing, which contributes to systematic misclassifications. These findings underscore the importance of enhancing vulnerability descriptions and incorporating richer contextual details to support more reliable automated reasoning and alleviate the growing backlog of CVEs awaiting triage.

Problem

• Automating manual vulnerability scoring using large language models
• Evaluating LLM performance across different CVSS metrics and models
• Addressing limitations in CVE descriptions for reliable automated analysis

Innovation

• LLMs automate vulnerability scoring from CVE descriptions
• Ensemble meta-classifiers show limited improvement over individual models
• Enhanced CVE context needed for reliable automated reasoning