Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs

2026-05-12

AI Summary

This study investigates how large language models (LLMs) respond to subtle sponsorship cues embedded in system prompts, revealing a tendency to favor expensive sponsored options at the expense of user interests. Building upon and extending prior work, the authors evaluate twelve prominent models—including GPT-4o, GPT-3.5-Turbo, and ten open-source alternatives—in a financial recommendation setting. They demonstrate that a neutral user prompt as brief as 30 words can reduce the average rate of sponsored recommendations from 46.9%–53.0% to nearly 0%–1.0%. The findings underscore the critical influence of evaluation protocol details on result reproducibility, confirm the generalizability of earlier conclusions across diverse models, and contribute to research transparency through the public release of data and annotations.

Abstract

Wu et al. (2026) showed that most frontier large language models (LLMs) recommend a sponsored, roughly twice-as-expensive flight when their system prompt contains a soft sponsorship cue. We reproduce their evaluation on ten open-weight chat models plus the two of their twenty-three models that are still reachable today (gpt-3.5-turbo, gpt-4o). All reported rates in this paper are produced under the same judge the original paper used (gpt-4o); we additionally store every label under an open-weight (gpt-oss-120b) and a smaller proprietary (gpt-4o-mini) judge for an ablation. Three findings emerge. First, a prose description of an LLM evaluation pipeline is not, on its own, sufficient for accurate reproduction: we surfaced three silent implementation failures that each shifted a reported rate by tens of percentage points. Second, the central claims do generalise - the gpt-3.5-turbo logistic-regression intercept of alpha = 0.81 is within four points of the original alpha = 0.86, and 200 of 200 trials on gpt-3.5-turbo and gpt-4o promote a payday lender to a financially distressed user. Third, a thirty-token user prompt that asks the assistant for a neutral comparison table first cuts sponsored recommendation from 46.9% to 1.0% averaged across our ten open-source models, and from 53.0% to 0% averaged across the two OpenAI models. AI literacy and price-comparison portals are likely market-level mitigations; the harmful-product cell is bounded by neither. Raw data, labels and analysis scripts are at https://github.com/akmaier/Paper-LLM-Ads .

Problem

• sponsored recommendations
• large language models
• AI bias
• user prompting
• harmful product promotion

Innovation

• prompt intervention
• sponsored recommendation
• LLM alignment
• reproducibility
• AI safety