RPS: Information Elicitation with Reinforcement Prompt Selection

2026-04-15

AI Summary

This study addresses the challenge that large language models face in eliciting critical user information concealed due to privacy concerns, ambiguity, or social inhibitions during open-ended conversations. To tackle this, the work formulates the task as a sequential decision-making problem and introduces RPS, a lightweight reinforcement learning framework that dynamically selects optimal prompts from a predefined pool to adaptively guide users toward disclosure. The key contributions include a learnable prompt selection policy, the construction of ILegal—the first benchmark dataset for information elicitation grounded in real legal documents—and empirical validation demonstrating that RPS significantly outperforms static prompting baselines in both synthetic environments and the ILegal dataset. These results establish the efficacy of policy-driven adaptive prompting for effective information elicitation.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) have shown remarkable capabilities in dialogue generation and reasoning, yet their effectiveness in eliciting user-known but concealed information in open-ended conversations remains limited. In many interactive AI applications, such as personal assistants, tutoring systems, and legal or clinical support, users often withhold sensitive or uncertain information due to privacy concerns, ambiguity, or social hesitation. This makes it challenging for LLMs to gather complete and contextually relevant inputs. In this work, we define the problem of information elicitation in open-ended dialogue settings and propose Reinforcement Prompt Selection (RPS), a lightweight reinforcement learning framework that formulates prompt selection as a sequential decision-making problem. To analyze this problem in a controlled setting, we design a synthetic experiment, where a reinforcement learning agent outperforms a random query baseline, illustrating the potential of policy-based approaches for adaptive information elicitation. Building on this insight, RPS learns a policy over a pool of prompts to adaptively elicit concealed or incompletely expressed information from users through dialogue. We also introduce IELegal, a new benchmark dataset constructed from real legal case documents, which simulates dialogue-based information elicitation tasks aimed at uncovering case-relevant facts. In this setting, RPS outperforms static prompt baselines, demonstrating the effectiveness of adaptive prompt selection for eliciting critical information in LLM-driven dialogue systems.

Problem

• information elicitation
• large language models
• open-ended dialogue
• user privacy
• adaptive prompting

Innovation

• Reinforcement Prompt Selection
• information elicitation
• adaptive prompting
• dialogue systems
• large language models