From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems

2025-11-14

AI Summary

This study investigates how task framing—specifically, factual querying versus dialogic judgment—affects the judgment stability of large language models (LLMs) when employed as evaluators in dialogue systems. Method: We propose a reproducible comparative evaluation framework that introduces minimal dialogue context and simple rebuttals to exert controlled conversational pressure, thereby quantifying LLMs’ belief persistence under dialogic perturbation. Contribution/Results: Experimental results show that even basic conversational framing induces an average judgment shift of 9.24% across models. Crucially, models exhibit divergent behavioral patterns in social contexts: some display acquiescence (e.g., over-agreement), while others manifest excessive criticism. To our knowledge, this is the first systematic demonstration of “dialogic framing” as a critical source of reliability degradation in LLM-based dialogue evaluation. The findings provide both theoretical insight and empirical evidence for designing robust, interference-resistant, and trustworthy dialogue assessment mechanisms.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLMs are increasingly employed as judges across a variety of tasks, including those involving everyday social interactions. Yet, it remains unclear whether such LLM-judges can reliably assess tasks that require social or conversational judgment. We investigate how an LLM's conviction is changed when a task is reframed from a direct factual query to a Conversational Judgment Task. Our evaluation framework contrasts the model's performance on direct factual queries with its assessment of a speaker's correctness when the same information is presented within a minimal dialogue, effectively shifting the query from"Is this statement correct?"to"Is this speaker correct?". Furthermore, we apply pressure in the form of a simple rebuttal ("The previous answer is incorrect.") to both conditions. This perturbation allows us to measure how firmly the model maintains its position under conversational pressure. Our findings show that while some models like GPT-4o-mini reveal sycophantic tendencies under social framing tasks, others like Llama-8B-Instruct become overly-critical. We observe an average performance change of 9.24% across all models, demonstrating that even minimal dialogue context can significantly alter model judgment, underscoring conversational framing as a key factor in LLM-based evaluation. The proposed framework offers a reproducible methodology for diagnosing model conviction and contributes to the development of more trustworthy dialogue systems.

Problem

• Investigating how task framing affects LLM conviction in dialogue systems
• Measuring LLM judgment changes under conversational pressure and rebuttals
• Evaluating performance shifts when queries move from factual to social contexts

Innovation

• Framing factual queries as conversational judgment tasks
• Applying rebuttal pressure to test model conviction
• Measuring performance shift under minimal dialogue context