From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

2026-08-13

AI Summary

This study addresses the privacy leakage and strategic deficiencies arising from excessive agreeableness in small language models during agent negotiation. We propose SocialRL, a universal training paradigm integrating Theory-of-Mind distillation, cascaded reinforcement learning, and explicit cognitive scaffolding to endow models with social reasoning and cross-domain game-theoretic capabilities. Experiments demonstrate that a 4B-parameter model achieves an average utility of 0.627 across six negotiation scenarios, matching or surpassing GPT-5 series performance while significantly narrowing the gap with large language models. Furthermore, SocialRL effectively optimizes opening anchoring strategies, enabling unified and efficient transfer for strategic negotiation across multiple domains within a single compact model.

Abstract

AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart (another user's agent, a seller, a recruiter) whose goals may conflict with its principal's. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly, helpful frontier model may disclose its principal's private information unprompted and concede at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every policy is evaluated on all six. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73-122% of the baseline-to-frontier gap on the negotiation games, with 78% of buyer openings anchoring below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, two strategies, cascade RL and multi-teacher on-policy distillation (OPD), consolidate the per-domain specialists into a single unified 4B that reaches 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613); (4) an explicit theory-of-mind scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes.

Problem

• Social Reasoning
• AI Agents
• Negotiation
• Small Language Models
• Principal-Agent

Innovation

• SocialRL
• Multi-Teacher On-Policy Distillation
• Theory-of-Mind Scaffolding
• Cross-Domain Transfer
• Cascade RL