🤖 AI Summary
Existing automatic evaluation methods struggle to effectively detect long-range inconsistencies in multi-turn dialogues, such as contradictions, topic drift, and entity conflicts. This work proposes the first stateful evaluation framework based on an incremental Semantic Knowledge Graph (SKG), which dynamically tracks entities, relations, and commitments throughout a dialogue by leveraging structured triple extraction and graph embeddings. The approach integrates three complementary signals—local relevance, historical consistency, and logical coherence—to explicitly identify cross-turn contradictions without relying on large language models or natural language inference systems, thereby producing auditable evaluation certificates. Experimental results demonstrate that the framework significantly improves correlation with human judgments across multiple benchmarks, substantially enhances detection of long-range inconsistencies, and yields deterministic scores under fixed inputs.
📝 Abstract
Evaluating multi-turn dialogue systems remains challenging because response quality depends not only on the current prompt, but also on previously established entities, claims, and conversational commitments. Existing automatic evaluators, including LLM-as-a-judge frameworks and embedding-based metrics, largely rely on flat or turn-isolated representations, making them less effective at detecting long-range issues such as contradiction, topic drift, and entity inconsistency. To address this, we propose SKG-Eval, a quasi-deterministic and interpretable framework that models dialogue as an evolving Semantic Knowledge Graph (SKG) of entities, relations, and commitments across turns. The framework incrementally updates the graph through structured triple extraction and computes three complementary signals: (i) local relevance, measuring alignment with the current prompt and optional reference; (ii) historical consistency, evaluating how newly introduced information connects to prior conversational context using graph-based and embedding-driven signals; and (iii) logical coherence, assessed by a geometric contradiction engine that detects cross-turn conflicts without relying on NLI models or LLM judges. These signals are adaptively fused and aggregated into a length-invariant session score via recency-weighted trend analysis. Across multiple benchmarks, SKG-Eval achieves higher correlation with human judgments and substantially improves detection of long-range inconsistencies in extended conversations. In addition, the framework produces explicit contradiction certificates and deterministic scores for fixed inputs, enabling reproducible and auditable evaluation. Overall, our results suggest that structured externalized state tracking through semantic knowledge graphs provides a scalable alternative to implicit reasoning in LLM-based dialogue evaluators.