Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting

2025-08-30

AI Summary

In the CPDC 2025 API Track, role-playing dialogue agents suffer from two critical issues: verbose responses (over-speaking) and improper tool usage (e.g., invalid or premature function calls), undermining role consistency and behavioral controllability. To address these, we propose Rule-based Role Prompting (RRP), a novel method that introduces structured role cards and scene contracts, while strictly constraining the timing and validity of function calls. Compared to baseline prompting, handcrafted prompts, and automated prompt optimization (APO), RRP significantly improves role fidelity and tool-use accuracy—raising the overall score from 0.519 to 0.571, outperforming zero-shot baselines and multiple advanced prompting techniques. We open-source our optimal prompt templates and APO toolkit, establishing a reproducible benchmark and practical framework for controllable role-playing dialogue research.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This report investigates approaches for prompting a tool-augmented large language model (LLM) to act as a role-playing dialogue agent in the API track of the Commonsense Persona-grounded Dialogue Challenge (CPDC) 2025. In this setting, dialogue agents often produce overly long in-character responses (over-speaking) while failing to use tools effectively according to the persona (under-acting), such as generating function calls that do not exist or making unnecessary tool calls before answering. We explore four prompting approaches to address these issues: 1) basic role prompting, 2) human-crafted role prompting, 3) automatic prompt optimization (APO), and 4) rule-based role prompting. The rule-based role prompting (RRP) approach achieved the best performance through two novel techniques--character-card/scene-contract design and strict enforcement of function calling--which led to an overall score of 0.571, improving on the zero-shot baseline score of 0.519. These findings demonstrate that RRP design can substantially improve the effectiveness and reliability of role-playing dialogue agents compared with more elaborate methods such as APO. To support future efforts in developing persona prompts, we are open-sourcing all of our best-performing prompts and the APO tool. Source code is available at https://github.com/scb-10x/apo.

Problem

• Reducing overly long responses in role-play dialogue agents
• Improving tool usage accuracy for persona-consistent actions
• Optimizing prompt design to prevent unnecessary function calls

Innovation

• Rule-based role prompting with character-card design
• Strict enforcement of function calling protocols
• Automatic prompt optimization for dialogue agents