Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

2026-08-17

AI Summary

This study addresses the exploration-exploitation imbalance caused by semantic priors in large language model decision-making by proposing a Semantic Multi-Armed Bandit framework. This approach formally quantifies how the alignment between linguistic labels and reward structures influences exploration strategies. Through in-context learning and inductive bias analysis, we reveal the bias effects inherent in label semantics and reward signals. Empirical results demonstrate that semantically consistent labels significantly enhance decision performance, whereas mismatches cause severe degradation. Furthermore, negative rewards elicit greater exploration than positive ones, validating scale biases present in pretraining data. Collectively, this work offers novel insights into understanding and optimizing the decision-making behaviors of large language models.

Abstract

Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.

Problem

• Semantic Bandits
• Exploration-Exploitation
• Semantic Priors
• Large Language Models
• Decision-Making Agents

Innovation

• Semantic Bandits
• Exploration-Exploitation
• Semantic Priors
• Large Language Models
• Inductive Bias