Secret Breach Detection in Source Code with Large Language Models

2025-04-26

AI Summary

Existing approaches for detecting sensitive information (e.g., API keys, tokens) in source code suffer from high false-positive rates and insufficient contextual understanding. Method: This paper proposes a hybrid method combining regex-based pre-filtering with large language model (LLM)-based classification. It leverages LoRA-finetuned open-source LLMs—specifically LLaMA-3.1 8B and Mistral-7B—for multi-class key identification, supporting zero-shot and few-shot prompting as well as lightweight local deployment. Contribution/Results: To the best of our knowledge, this is the first systematic validation of open-weight LLMs fine-tuned via LoRA for sensitive token classification. Experiments show that the fine-tuned LLaMA-3.1 8B achieves a binary classification F1-score of 0.9852, while Mistral-7B attains a multi-class accuracy of 0.982—both significantly outperforming traditional regex baselines while maintaining high recall. The approach balances detection accuracy, data privacy, and computational cost.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Background: Leaking sensitive information, such as API keys, tokens, and credentials, in source code remains a persistent security threat. Traditional regex and entropy-based tools often generate high false positives due to limited contextual understanding. Aims: This work aims to enhance secret detection in source code using large language models (LLMs), reducing false positives while maintaining high recall. We also evaluate the feasibility of using fine-tuned, smaller models for local deployment. Method: We propose a hybrid approach combining regex-based candidate extraction with LLM-based classification. We evaluate pre-trained and fine-tuned variants of various Large Language Models on a benchmark dataset from 818 GitHub repositories. Various prompting strategies and efficient fine-tuning methods are employed for both binary and multiclass classification. Results: The fine-tuned LLaMA-3.1 8B model achieved an F1-score of 0.9852 in binary classification, outperforming regex-only baselines. For multiclass classification, Mistral-7B reached 0.982 accuracy. Fine-tuning significantly improved performance across all models. Conclusions: Fine-tuned LLMs offer an effective and scalable solution for secret detection, greatly reducing false positives. Open-source models provide a practical alternative to commercial APIs, enabling secure and cost-efficient deployment in development workflows.

Problem

• Detect sensitive information leaks in source code
• Reduce false positives in secret detection using LLMs
• Evaluate fine-tuned small models for local deployment

Innovation

• Hybrid regex-LLM approach for secret detection
• Fine-tuned LLaMA-3.1 8B achieves high F1-score
• Open-source models enable secure local deployment