Neural network embeddings recover value dimensions from psychometric survey items on par with human data

2025-09-29

AI Summary

This study addresses two longstanding challenges in psychometrics: reliance on labor-intensive human scoring and difficulty modeling negative interdimensional correlations. We propose SQuID (Semantic Questionnaire Item Discovery), a novel method that directly leverages raw semantic embeddings from un-fine-tuned large language models (LLMs) to represent questionnaire items, then recovers and aligns latent value dimensions via embedding-difference analysis. Its key contribution is the first demonstration that purely unsupervised, pre-trained semantic embeddings—without domain-specific fine-tuning—can accurately reconstruct the negative correlational structure captured by human ratings. SQuID achieves high dimensional fidelity, explaining 55% of variance and yielding multidimensional scaling configurations highly consistent with human judgments (factor congruence coefficient > 0.92). The approach is cost-efficient, scalable across domains, and theoretically interpretable, establishing a neural-embedding-based paradigm for efficient, automated psychometric assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This study introduces "Survey and Questionnaire Item Embeddings Differentials" (SQuID), a novel methodological approach that enables neural network embeddings to effectively recover latent dimensions from psychometric survey items. We demonstrate that embeddings derived from large language models, when processed with SQuID, can recover the structure of human values obtained from human rater judgments on the Revised Portrait Value Questionnaire (PVQ-RR). Our experimental validation compares multiple embedding models across a number of evaluation metrics. Unlike previous approaches, SQuID successfully addresses the challenge of obtaining negative correlations between dimensions without requiring domain-specific fine-tuning. Quantitative analysis reveals that our embedding-based approach explains 55% of variance in dimension-dimension similarities compared to human data. Multidimensional scaling configurations from both types of data show fair factor congruence coefficients and largely follow the underlying theory. These results demonstrate that semantic embeddings can effectively replicate psychometric structures previously established through extensive human surveys. The approach offers substantial advantages in cost, scalability and flexibility while maintaining comparable quality to traditional methods. Our findings have significant implications for psychometrics and social science research, providing a complementary methodology that could expand the scope of human behavior and experience represented in measurement tools.

Problem

• Recovering latent dimensions from psychometric survey items using neural embeddings
• Addressing negative correlation challenges without domain-specific fine-tuning
• Explaining 55% variance in dimension similarities compared to human data

Innovation

• SQuID method enables neural embeddings recovery
• LLM embeddings replicate human value structures
• No domain fine-tuning for negative correlations