GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

2026-05-27

AI Summary

This work addresses the lack of standardized evaluation criteria for ranking manipulation in generative search engines, which hinders fair comparison of attack efficacy and stealth across methods. The authors propose GEO-Bench, the first standardized benchmark for generative engine optimization, integrating black-box prompt attacks, white-box gradient-based attacks, and ten white-hat C-SEO strategies. Evaluations are conducted on a fixed ranking model (Llama-3.1-8B-Instruct) across multiple datasets, using effectiveness metrics such as NRG and Success@α alongside stealth metrics including keyword violation rate and perplexity ratio. Experimental results reveal a pronounced trade-off between effectiveness and stealth across attack paradigms, demonstrate that black-box rewriting can outperform gradient-based attacks in certain scenarios, and indicate that model access privileges are not the primary determinant of attack strength.

Abstract

Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a growing concern for fairness and information integrity. Research on generative engine optimization (GEO) has produced many manipulation methods, but each is evaluated on its own dataset with its own metrics, so their relative strength and detectability stay unclear. We present GEO-Bench, a benchmark that evaluates GEO ranking-manipulation attacks under one protocol. It unifies black-box prompt-based attacks (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten white-hat C-SEO strategies. We score every method on five datasets against a fixed open-weight ranker (Llama-3.1-8B-Instruct), using metrics for both effectiveness (NRG, Success@α, Promote@α) and stealth (keyword violation rate, perplexity ratio). Our evaluation shows that effectiveness and stealth trade off across adversarial attacks, that black-box content rewriting matches or exceeds gradient-based attacks on rank promotion while producing more fluent text and can evade both keyword- and perplexity-based detection on some domains, and that the access model does not predict attack strength. By standardizing datasets, attack implementations, and metrics, GEO-Bench enables the first direct comparison across these attack paradigms and supports the development of detection methods.

Problem

• ranking manipulation
• generative engine optimization
• benchmark
• adversarial attacks
• information integrity

Innovation

• Generative Engine Optimization
• Ranking Manipulation
• Benchmarking
• Adversarial Attacks
• Stealth Detection