🤖 AI Summary
This study addresses the critical gap in systematic evaluation of large language models (LLMs) against the OWASP Top 10 security risks specific to LLM applications. Leveraging the OWASP Top 10 for LLM Applications framework, the authors conduct the first comprehensive assessment of five standard Llama models and five Llama Guard variants. They construct a benchmark comprising 100 adversarial prompts spanning all ten vulnerability categories and evaluate threat detection accuracy, response safety, and computational overhead on the FABRIC platform using NVIDIA A30 GPUs. Results reveal that the lightweight specialized model Llama-Guard-3-1B achieves the best performance with a 76% detection rate and 0.165-second latency, whereas base models such as Llama-3.1-8B exhibit 0% detection, indicating an inverse relationship between model scale and security efficacy. The benchmark dataset, including adversarial prompts and attack metadata, is publicly released.
📝 Abstract
As large language models (LLMs) move from research prototypes to enterprise systems, their security vulnerabilities pose serious risks to data privacy and system integrity. This study benchmarks various Llama model variants against the OWASP Top 10 for LLM Applications framework, evaluating threat detection accuracy, response safety, and computational overhead. Using the FABRIC testbed with NVIDIA A30 GPUs, we tested five standard Llama models and five Llama Guard variants on 100 adversarial prompts covering ten vulnerability categories. Our results reveal significant differences in security performance: the compact Llama-Guard-3-1B model achieved the highest detection rate of 76% with minimal latency (0.165s per test), whereas base models such as Llama-3.1-8B failed to detect threats (0% accuracy) despite longer inference times (0.754s). We observe an inverse relationship between model size and security effectiveness, suggesting that smaller, specialized models often outperform larger general-purpose ones in security tasks. Additionally, we provide an open-source benchmark dataset including adversarial prompts, threat labels, and attack metadata to support reproducible research in AI security, [1].