Modeling Subjectivity in Cognitive Appraisal with Language Models

2025-03-14

AI Summary

This study addresses critical gaps in subjective modeling within NLP—particularly the underrepresentation of cognitive dimensions such as user confidence and interpersonal disagreement. Existing approaches often neglect personality traits and demographic variables, while post-hoc calibration methods exhibit limited efficacy. To bridge these gaps, we propose a dual-path framework integrating fine-tuning and prompt engineering, validated across diverse scenarios via quantitative and interpretable analyses. We provide the first systematic empirical evidence that personality (e.g., openness, agreeableness) and demographics (e.g., age, education) act as key moderating variables in subjective modeling. We further demonstrate the substantial failure of prevalent post-hoc calibration techniques. Finally, we release an open, reproducible benchmark and a theory-driven evaluation protocol, fostering deeper integration between NLP and cognitive science. Our work establishes a novel “human-factor-aware” paradigm for subjective modeling grounded in cognitive realism.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

As the utilization of language models in interdisciplinary, human-centered studies grow, the expectation of model capabilities continues to evolve. Beyond excelling at conventional tasks, models are recently expected to perform well on user-centric measurements involving confidence and human (dis)agreement -- factors that reflect subjective preferences. While modeling of subjectivity plays an essential role in cognitive science and has been extensively studied, it remains under-explored within the NLP community. In light of this gap, we explore how language models can harness subjectivity by conducting comprehensive experiments and analysis across various scenarios using both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative experimental results indicate that existing post-hoc calibration approaches often fail to produce satisfactory results. However, our findings reveal that personality traits and demographical information are critical for measuring subjectivity. Furthermore, our in-depth analysis offers valuable insights for future research and development in the interdisciplinary studies of NLP and cognitive science.

Problem

• Modeling subjectivity in cognitive appraisal using language models.
• Exploring subjectivity measurement with personality traits and demographics.
• Improving post-hoc calibration for subjective preferences in NLP.

Innovation

• Utilizes fine-tuned and prompt-based LLMs
• Explores subjectivity in cognitive appraisal
• Highlights personality and demographic factors