FrameBench:A Language Understanding Benchmark Based on Frame Semantics

2026-09-03

AI Summary

研究通过构建基于框架语义的FrameBench基准，使用多选题测试模型在不同上下文中区分同一动词所激发的框架能力，以评估语言模型对文本隐含信息的理解。

Abstract

In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at https://github.com/SasanoLab/FrameBench.

Problem

• frame semantics
• large language models
• implicit enrichment

Innovation

• frame semantics
• language understanding benchmark
• semantic frames
• multi-choice questions