Improving LLM Personas via Rationalization with Psychological Scaffolds

2025-04-25

AI Summary

Existing LLM-based user profiling primarily relies on demographic attributes or historical behavioral data, failing to model the underlying cognitive motivations behind user judgments—thus limiting preference prediction accuracy. To address this, we propose a psychology-enhanced LLM persona construction framework that, for the first time, integrates computable Big Five personality traits and the Original World Beliefs theory into a structured psychological scaffold, explicitly modeling user reasoning via a rational generation mechanism. Our approach synergizes theory-of-mind modeling, structured prompt engineering, and multi-source rationality fusion. Empirical evaluation on public opinion and movie preference prediction tasks demonstrates significant improvements over state-of-the-art baselines. Moreover, the generated rational explanations achieve cognitive plausibility comparable to human-authored ones. This work validates the effectiveness and novelty of theory-driven modeling in advancing LLMs’ capacity for deep user understanding.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Language models prompted with a user description or persona can predict a user's preferences and opinions, but existing approaches to building personas -- based solely on a user's demographic attributes and/or prior judgments -- fail to capture the underlying reasoning behind said user judgments. We introduce PB&J (Psychology of Behavior and Judgments), a framework that improves LLM personas by incorporating rationales of why a user might make specific judgments. These rationales are LLM-generated, and aim to reason about a user's behavior on the basis of their experiences, personality traits or beliefs. This is done using psychological scaffolds -- structured frameworks grounded in theories such as the Big 5 Personality Traits and Primal World Beliefs -- that help provide structure to the generated rationales. Experiments on public opinion and movie preference prediction tasks demonstrate that LLM personas augmented with PB&J rationales consistently outperform methods using only a user's demographics and/or judgments. Additionally, LLM personas constructed using scaffolds describing user beliefs perform competitively with those using human-written rationales.

Problem

• Enhance LLM personas by adding rationale for user judgments
• Use psychological scaffolds to structure generated rationales
• Improve prediction accuracy over demographic-only methods

Innovation

• Incorporates LLM-generated rationales for user judgments
• Uses psychological scaffolds like Big 5 Traits
• Enhances personas via belief and experience reasoning