Problem
Research questions and friction points this paper is trying to address.
Large Language Models
Contextual Understanding
Evaluation Metrics
Question Answering
Knowledge Graph
Evaluation of Contextual Understanding in Large Language Models
🤖 AI Summary
本文提出了一种基于知识图谱的评估框架S3KG，旨在更准确地衡量大型语言模型在问答任务中的上下文理解能力。
📝 Abstract
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.
Innovation
Methods, ideas, or system contributions that make the work stand out.
knowledge graph-based evaluation
Semantic Structural Similarity for KGs (S3KG)
reasoning errors
🔎 Similar Papers
No similar papers found.
