GenXSS: an AI-Driven Framework for Automated Detection of XSS Attacks in WAFs

2025-04-11

AI Summary

Traditional WAFs struggle to detect highly obfuscated XSS attacks due to their reliance on manually curated, static rule sets and inherent inability to adapt to novel evasion techniques. This paper introduces the first end-to-end XSS attack-defense closed-loop framework powered by large language models (LLMs)—specifically GPT-4o and Gemini Pro—that integrates in-context learning with dynamic payload validation to jointly enable malicious payload generation, black-box WAF evasion testing, and reverse synthesis of deployable protection rules. Its key innovation lies in automating the full “generate–test–defend” pipeline using LLMs without requiring white-box WAF access. Experimentally, the framework generated 264 syntactically valid XSS payloads, 83% of which were functionally effective; 80% successfully bypassed the OWASP ModSecurity rule set. Moreover, just 15 automatically synthesized rules achieved 86% interception coverage against known evasion variants.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The increasing reliance on web services has led to a rise in cybersecurity threats, particularly Cross-Site Scripting (XSS) attacks, which target client-side layers of web applications by injecting malicious scripts. Traditional Web Application Firewalls (WAFs) struggle to detect highly obfuscated and complex attacks, as their rules require manual updates. This paper presents a novel generative AI framework that leverages Large Language Models (LLMs) to enhance XSS mitigation. The framework achieves two primary objectives: (1) generating sophisticated and syntactically validated XSS payloads using in-context learning, and (2) automating defense mechanisms by testing these attacks against a vulnerable application secured by a WAF, classifying bypassing attacks, and generating effective WAF security rules. Experimental results using GPT-4o demonstrate the framework's effectiveness generating 264 XSS payloads, 83% of which were validated, with 80% bypassing ModSecurity WAF equipped with an industry standard security rule set developed by the Open Web Application Security Project (OWASP) to protect against web vulnerabilities. Through rule generation, 86% of previously successful attacks were blocked using only 15 new rules. In comparison, Google Gemini Pro achieved a lower bypass rate of 63%, highlighting performance differences across LLMs.

Problem

• Detecting obfuscated XSS attacks bypassing traditional WAFs
• Automating XSS payload generation and validation using AI
• Improving WAF rules to block previously successful attacks

Innovation

• Leverages LLMs for XSS payload generation
• Automates WAF defense mechanism testing
• Generates effective WAF security rules