Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

2026-07-15

AI Summary

This work addresses the limitations of existing vision-language model evaluations, which predominantly rely on static benchmarks and fail to capture contextually accumulated hallucinations in real-world interactive settings. To this end, the authors propose CEDI, a novel evaluation framework that introduces multi-turn dynamic interaction and state-space navigation into the assessment process. CEDI orchestrates semi-structured dialogues among a task-graph-guided model, an automated examiner, and a scorer, integrating graph-structured task representations, adversarial probes, contextual state tracking, and automated scoring to dynamically probe model behavior. Experimental results demonstrate that CEDI substantially outperforms conventional static methods, more effectively exposing model vulnerabilities in complex scenarios such as long-context reasoning, premise rejection, and refusal to answer, thereby yielding evaluation outcomes that better reflect real-world performance.

Abstract

Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain. This gap stems from the fundamental misalignment between benchmarks in controlled, static settings and the dynamic, interactive, and contextualized nature of real-world applications. To bridge this gap, we propose CEDI (Contextualized Evaluations of MLLMs through Dynamic, multi-round Interactions), a framework that recasts evaluation as a three-party interaction between an evaluatee model, an automated examiner, and a grader. The examiner conducts multi-turn, semi-structured conversation guided by a graph-based representation of the task. By navigating state-space transitions, CEDI deploys diverse strategies, from clarification requests to adversarial probes, to elicit performance evidence. We apply CEDI to visual hallucinations. Empirical results across multiple models, diverse settings, datasets, and domains show that contextualized, interactive evaluations reveal not only significantly more hallucinations than conventional static evaluation but also ones that more closely resemble those arising in practical use cases. We further show that hallucinations often accumulate over long contexts, through self-reinforcing dialogue history, and models are particularly vulnerable to questions requiring premise rejection or refusal. Together, these findings highlight CEDI as a step toward realistic, systematic, and ecologically valid assessments of MLLMs' capabilities. Code is available at github.com/williamium3000/cedi.

Problem

• Vision Language Models
• Evaluation
• Dynamic Interaction
• Visual Hallucination
• Contextualized Assessment

Innovation

• Contextualized Evaluation
• Dynamic Interaction
• Multi-turn Dialogue
• Visual Hallucination
• Multi-modal LLMs