ALIGNS: Unlocking nomological networks in psychological measurement through a large language model

2025-09-10

AI Summary

Nomological network construction in psychometrics has long relied on labor-intensive manual efforts, impeding systematic validation of construct validity and limiting sensitivity in clinical trials and scientific rigor in public policy. Method: This study pioneers the integration of large language models (LLMs) into foundational construct validity assessment, training on empirically validated questionnaire data and combining classification accuracy evaluation with expert review to automate the construction of three comprehensive, cross-domain nomological networks spanning psychology, medicine, and social policy. Contribution/Results: The resulting interpretable network comprises over 550,000 indicators, revealing emotion distress as unidimensional and identifying four novel latent dimensions of child temperament. This work substantially enhances the scalability, systematicity, and accessibility of nomological network construction, establishing the first LLM-driven, evidence-based framework for construct validity validation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Psychological measurement is critical to many disciplines. Despite advances in measurement, building nomological networks, theoretical maps of how concepts and measures relate to establish validity, remains a challenge 70 years after Cronbach and Meehl proposed them as fundamental to validation. This limitation has practical consequences: clinical trials may fail to detect treatment effects, and public policy may target the wrong outcomes. We introduce Analysis of Latent Indicators to Generate Nomological Structures (ALIGNS), a large language model-based system trained with validated questionnaire measures. ALIGNS provides three comprehensive nomological networks containing over 550,000 indicators across psychology, medicine, social policy, and other fields. This represents the first application of large language models to solve a foundational problem in measurement validation. We report classification accuracy tests used to develop the model, as well as three evaluations. In the first evaluation, the widely used NIH PROMIS anxiety and depression instruments are shown to converge into a single dimension of emotional distress. The second evaluation examines child temperament measures and identifies four potential dimensions not captured by current frameworks, and questions one existing dimension. The third evaluation, an applicability check, engages expert psychometricians who assess the system's importance, accessibility, and suitability. ALIGNS is freely available at nomologicalnetwork.org, complementing traditional validation methods with large-scale nomological analysis.

Problem

• Building nomological networks for psychological measurement validation
• Addressing limitations in detecting treatment effects and policy outcomes
• Applying large language models to solve foundational validation challenges

Innovation

• Uses large language model trained with questionnaires
• Generates nomological networks with 550000 indicators
• Provides free web platform for validation analysis