Chasing Shadows: Pitfalls in LLM Security Research

2025-12-10

AI Summary

This work identifies nine methodological pitfalls inherent to large language model (LLM) applications in security research—spanning data collection, pretraining, fine-tuning, prompt engineering, and evaluation—that critically undermine reproducibility, rigor, and assessment validity. Through a multi-pronged empirical approach—including bibliometric analysis of 72 top-tier conference papers, cross-paper defect pattern mining, four controlled case studies, and reproducibility stress testing—we systematically discover and validate these LLM-specific issues for the first time. Our analysis reveals that 100% of examined papers exhibit at least one pitfall, yet only 15.7% explicitly acknowledge them. We propose an actionable, practice-oriented methodological guideline to address these gaps, bridging the critical void between methodological critique and practical improvement in LLM security research. This work lays foundational groundwork for establishing a trustworthy, scientifically robust AI security research paradigm.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) are increasingly prevalent in security research. Their unique characteristics, however, introduce challenges that undermine established paradigms of reproducibility, rigor, and evaluation. Prior work has identified common pitfalls in traditional machine learning research, but these studies predate the advent of LLMs. In this paper, we identify emph{nine} common pitfalls that have become (more) relevant with the emergence of LLMs and that can compromise the validity of research involving them. These pitfalls span the entire computation process, from data collection, pre-training, and fine-tuning to prompting and evaluation. We assess the prevalence of these pitfalls across all 72 peer-reviewed papers published at leading Security and Software Engineering venues between 2023 and 2024. We find that every paper contains at least one pitfall, and each pitfall appears in multiple papers. Yet only 15.7% of the present pitfalls were explicitly discussed, suggesting that the majority remain unrecognized. To understand their practical impact, we conduct four empirical case studies showing how individual pitfalls can mislead evaluation, inflate performance, or impair reproducibility. Based on our findings, we offer actionable guidelines to support the community in future work.

Problem

• Identifies pitfalls in LLM security research validity
• Assesses prevalence of pitfalls in recent peer-reviewed papers
• Provides guidelines to improve reproducibility and evaluation rigor

Innovation

• Identifies nine pitfalls in LLM security research
• Assesses pitfalls across 72 peer-reviewed papers empirically
• Provides actionable guidelines to address these pitfalls