Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA

2026-05-24

AI Summary

This study addresses the persistent challenge in multi-turn question answering where user intent is ambiguous, particularly the difficulty in generating accurate answers even after clarification. The work decouples this problem into two phases: clarification strategy and post-clarification answering. It proposes a supervised fine-tuning approach to optimize clarification strategies and introduces the PACIFIC evaluation framework to systematically analyze performance across both phases. Findings reveal that while clarification strategies can be effectively improved through fine-tuning, models still exhibit substantially low answer accuracy after correctly executing clarifications, highlighting that accurately interpreting users’ clarification responses constitutes a critical bottleneck in achieving intent alignment.

Abstract

Pluralistic alignment requires systems to adapt to diverse user values, communication styles, and contextual assumptions. We believe that a foundational prerequisite for such alignment enabling accurate preference elicitation from people when their intent is under-specified or ambiguous. We study the problem of preference elicitation in multi-turn question answering by decomposing the problem into two components: a \textbf{clarification policy}, which decides whether to ask a clarifying question or answer directly, and \textbf{post-clarification answering}, which produces the correct final answer once the missing information is provided. We show, using the PACIFIC benchmark, that supervised fine-tuning rapidly improves the clarification policy, however, final answer accuracy remains substantially lower even when the model takes the correct action. This gap indicates that understanding and correctly interpreting the user's response is the critical gap in multi-turn question-answering systems.

Problem

• preference elicitation
• multi-turn QA
• post-clarification answering
• clarification policy
• pluralistic alignment

Innovation

• preference elicitation
• multi-turn QA
• clarification policy
• post-clarification answering
• pluralistic alignment