Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards

2026-04-10

AI Summary

This work addresses the limited strategic capabilities of large language models in bilateral price negotiation under incomplete information. The authors propose a reinforcement learning framework grounded in verifiable rewards, training a 30-billion-parameter buyer agent against constrained sellers over real-world product distributions, with economic surplus maximization and budget adherence serving directly as reward signals. The study uncovers a four-stage evolution in the agent’s negotiation strategy, progressing from basic bargaining to sophisticated persuasive tactics. The trained agent significantly outperforms state-of-the-art models more than ten times its size in surplus extraction and demonstrates robust generalization to unseen, highly adversarial sellers.

Technology Category

• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Adversarial Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The recent advancement of Large Language Models (LLMs) has established their potential as autonomous interactive agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic evolution. The agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile, adversarial seller personas.

Problem

• Large Language Models
• Negotiation
• Strategic Games
• Incomplete Information
• Autonomous Agents

Innovation

• Reinforcement Learning from Verifiable Rewards
• strategic negotiation
• economic surplus maximization
• budget-constrained LLM agents
• four-phase strategic evolution