DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

2026-05-22

AI Summary

This work addresses the instability of large language models in reasoning about conversational implicatures that rely on social and contextual cues, a challenge exacerbated by the lack of targeted evaluation frameworks. The authors propose DRInQ, the first controllable benchmark specifically designed for question-based conversational implicature, which fixes surface question forms while systematically varying contextual information. Leveraging a semi-automated pipeline, DRInQ generates large-scale question–context–explanation triplets. The study reveals a marked asymmetry in model performance: while models struggle to generate plausible pragmatic scenarios, they show greater competence in inferring implied meanings when provided with appropriate contexts. Structured prompting significantly improves alignment between smaller models and human judgments, and complementary strengths are observed between humans and models in context creation.

Abstract

Human conversation relies heavily on conversational implicature, in which speakers convey meanings that are suggested rather than explicitly stated. Although recent large language models exhibit strong conversational fluency, they remain unreliable when interpretation depends on reasoning that integrates social and contextual cues, a process rarely articulated in text. We introduce DRinQ, a benchmark for evaluating pragmatic reasoning about conversational implicature in question utterances, designed to isolate pragmatic variation while holding each question's surface form fixed. To support scalable evaluation, we propose a semi-automated pipeline that produces question-context-interpretation instances with systematic variation. Across evaluations, we find a consistent generation-inference asymmetry: while state-of-the-art models can generate plausible pragmatic scenarios when guided, they often fail to recover the intended implication at inference time. For smaller models, structured prompting improves alignment with human judgments. A comparative writing study further reveals complementary strengths: human authors tend to produce safer, predictable contexts, whereas models generate varied scenarios with interpretations that sometimes exceed contextual support. These findings highlight persistent challenges in modeling conversational implicature and motivate more context-sensitive evaluation frameworks.

Problem

• conversational implicature
• pragmatic reasoning
• contextual cues
• language models
• evaluation benchmark

Innovation

• conversational implicature
• pragmatic reasoning
• controlled context variation
• benchmark evaluation
• generation-inference asymmetry