Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

2025-06-01

AI Summary

Large language models (LLMs) exhibit weak structured reasoning and unreliable tool invocation when applied to cybersecurity tasks—particularly cryptography-focused Capture-the-Flag (CTF) challenges. Method: This paper introduces *random-crypto*, the first framework for generating randomized cryptographic CTF challenges, and develops a tool-augmented Llama-3.1-8B agent that iteratively writes and executes code within an isolated Python REPL environment. Crucially, it pioneers the application of Guided Reinforcement Prompt Optimization (GRPO) for end-to-end reinforcement fine-tuning of LLM agents on cryptographic CTF tasks. Contribution/Results: GRPO simultaneously improves tool-call reliability and code synthesis quality. On unseen *random-crypto* tasks, the agent achieves Pass@8 = 0.88 (+53 percentage points over baselines) and Majority@8 = 0.41. When transferred to *picoCTF*, performance improves by 13 percentage points, demonstrating strong generalization across cryptographic CTF domains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Security and Privacy: Applications of cryptography

Abstract

Large Language Models (LLMs) still struggle with the structured reasoning and tool-assisted computation needed for problem solving in cybersecurity applications. In this work, we introduce"random-crypto", a cryptographic Capture-the-Flag (CTF) challenge generator framework that we use to fine-tune a tool-augmented Llama-3.1-8B with Guided Reinforcement Prompt Optimisation (GRPO), allowing the agent to iteratively write and execute Python inside an isolated REPL. GRPO yields a +53% absolute jump in Pass@8 on unseen"random-crypto"tasks (0.35 ->0.88) and raises Majority@8 to 0.41. The fine-tuned agent also generalizes to an external dataset. On a subset of picoCTF cryptography problems, it improves Pass@8 by +13 pp. Ablations show the gains stem from more reliable tool invocation and code synthesis, rather than superficial prompt adaptation.

Problem

• Enhancing LLM reasoning for cybersecurity problem-solving
• Optimizing tool-augmented agents via reinforcement learning
• Improving code synthesis for cryptographic challenges

Innovation

• Reinforcement learning fine-tunes LLM on CTF challenges
• Guided Reinforcement Prompt Optimisation enhances performance
• Isolated REPL enables iterative Python execution and testing