When Intelligence Fails: An Empirical Study on Why LLMs Struggle with Password Cracking

2025-10-17

AI Summary

This study identifies the fundamental failure mechanisms of large language models (LLMs) in generating or cracking real-world passwords based on users’ structured attributes. Method: We systematically evaluate open-source LLMs—including TinyLLaMA, Falcon-RW-1B, and Flan-T5—on synthetic user profiles, measuring password generation accuracy under both plaintext and SHA-256 hash settings using Hit@1/5/10 metrics. Contribution/Results: All models achieve ≤1.5% Hit@10 accuracy—substantially underperforming conventional rule-based and combinatorial password cracking tools. Crucially, this work provides the first empirical evidence that pre-trained LLMs inherently lack cryptographic domain knowledge, user behavioral priors, and structured reasoning memory; consequently, they cannot effectively model real-world password distributions—even after unsupervised fine-tuning. Our findings establish critical methodological benchmarks and deliver a key caution regarding the applicability boundaries of LLMs in security-sensitive contexts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The remarkable capabilities of Large Language Models (LLMs) in natural language understanding and generation have sparked interest in their potential for cybersecurity applications, including password guessing. In this study, we conduct an empirical investigation into the efficacy of pre-trained LLMs for password cracking using synthetic user profiles. Specifically, we evaluate the performance of state-of-the-art open-source LLMs such as TinyLLaMA, Falcon-RW-1B, and Flan-T5 by prompting them to generate plausible passwords based on structured user attributes (e.g., name, birthdate, hobbies). Our results, measured using Hit@1, Hit@5, and Hit@10 metrics under both plaintext and SHA-256 hash comparisons, reveal consistently poor performance, with all models achieving less than 1.5% accuracy at Hit@10. In contrast, traditional rule-based and combinator-based cracking methods demonstrate significantly higher success rates. Through detailed analysis and visualization, we identify key limitations in the generative reasoning of LLMs when applied to the domain-specific task of password guessing. Our findings suggest that, despite their linguistic prowess, current LLMs lack the domain adaptation and memorization capabilities required for effective password inference, especially in the absence of supervised fine-tuning on leaked password datasets. This study provides critical insights into the limitations of LLMs in adversarial contexts and lays the groundwork for future efforts in secure, privacy-preserving, and robust password modeling.

Problem

• Evaluating LLMs' poor performance in password cracking tasks
• Identifying limitations in LLMs' domain adaptation for cybersecurity
• Comparing LLMs' weak password inference against traditional methods

Innovation

• Evaluating LLMs for password generation using user profiles
• Comparing LLM performance with traditional password cracking methods
• Identifying LLM limitations in domain adaptation for security