Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

2026-05-28

AI Summary

Existing evaluation frameworks struggle to measure the performance of large language model (LLM)-driven HTTP honeypots in a unified, scalable, and realistic manner. This work proposes the first systematic evaluation methodology, constructing an interactive environment comprising 16 real-world backend applications, integrating AI-powered attack agents with verifiable objectives, and incorporating control tasks to enable reproducible experiments. Centered on the LLM as the core honeypot engine, the framework quantitatively assesses performance through interaction duration, detection rates, and operational costs. Experimental results demonstrate that LLM-based honeypots significantly prolong attacker engagement, reduce the likelihood of detection by advanced models, and maintain cost efficiency against intelligent adversaries, while also revealing a trade-off between stealthiness and interaction depth in deception configurations.

Abstract

Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive honeypots configurations, and observe unique trade-offs, such as longer interactions at the cost of increased detection.

Problem

• LLM-powered honeypots
• evaluation framework
• HTTP honeypots
• cyber defense
• AI security

Innovation

• LLM-powered honeypots
• evaluation framework
• AI hacking agents
• high-interaction honeypots
• exploit goals