From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models

2026-09-11

AI Summary

本文提出ModelLog框架，通过符号约束明确模型行为的语义结构，评估语言模型的知识和推理能力，并将其与预训练的学习信号关联。

Abstract

While Large Language Models have improved rapidly, many fundamental questions remain about how to evaluate the knowledge and reasoning abilities they acquire, and how such evaluations relate to the learning signals used in pre-training. In this paper, we propose ModelLog, a declarative probabilistic framework for pre-training evaluation that makes the semantic structure of model behavior explicit and provides new formal tools for relating evaluation to learning. ModelLog specifies evaluation targets as symbolic constraints over token-level predictions and measures how strongly a model's distribution satisfies those constraints. We explore the framework through a new suite of tasks targeting negation, mutual exclusivity, and consistency, finding systematic failures that are difficult to characterize through token likelihood or answer accuracy alone. We further show that these evaluation scores can also be interpreted as losses, whose gradients reflect logical strength, informativeness, and variable-level sensitivity. This links evaluation and learning through a shared semantics, suggesting evaluation methods that diagnose model behavior while also helping to clarify the semantic structure of learning.

Problem

• Large Language Models
• evaluation
• knowledge and reasoning abilities
• pre-training
• learning signals

Innovation

• ModelLog
• declarative probabilistic framework
• symbolic constraints
• token-level predictions
• evaluation and learning