Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing

2026-08-27

AI Summary

本文通过结合网络心理测量和改进的Ising模型，利用语义相似性和全数据集解决大规模评估中构建反应项评分一致性估计问题。

Abstract

Scoring consistency for constructed-response items in large-scale assessments is typically estimated through double-scoring, which uses small samples and assumes independence among responses. We present an integrated framework combining network psychometrics with the Linguistic-Integrated Reliability Audit (LiRA) via a modified Ising model. The model defines a joint distribution over binary correctness labels with pairwise interactions set to the cosine similarity of sentence embeddings and a global bias parameter for item difficulty. LiRA's weighted majority voting over semantic neighborhoods is shown to approximate the conditional logistic distributions of this Ising model. The parametric framework supports benchmark score generation, uncertainty quantification, and missing label imputation; parameters are estimated by maximum pseudo-likelihood. The approach uses the full dataset without requiring extensive double-scoring, accounts for semantic dependencies, and provides diagnostics for rater inconsistencies. The integration of LiRA's scalable methodology with a probabilistic graphical model offers a comprehensive tool for reliability assessment in international assessments such as PIRLS, PISA, and TIMSS.

Problem

• scoring consistency
• constructed-response items
• large-scale assessments
• double-scoring

Innovation

• network psychometrics
• Ising model
• Linguistic-Integrated Reliability Audit (LiRA)
• cosine similarity
• semantic dependencies