Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts

2025-12-09

AI Summary

Existing personality detection paradigms—mapping posts to user embeddings and then to labels—are hindered by label scarcity and ambiguous semantic alignment. This paper proposes a scale-driven paradigm leveraging large language models’ (LLMs) role-playing capability to simulate users’ responses to standardized psychological instruments (e.g., MBTI), transforming free-text posts into interpretable, scale-anchored linguistic evidence. Methodologically, we explicitly inject psychological knowledge, design a question-conditioned Mixture-of-Experts (MoE) architecture for joint routing of posts and scale items with fine-grained supervision, and integrate interpretable answer vectors within a multi-task learning framework. Evaluated on two real-world datasets, our approach significantly outperforms state-of-the-art methods: it achieves a 15.41% accuracy gain on the Kaggle dataset, demonstrating both superior performance and enhanced psychological interpretability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Understanding human personality is crucial for web applications such as personalized recommendation and mental health assessment. Existing studies on personality detection predominantly adopt a"posts ->user vector ->labels"modeling paradigm, which encodes social media posts into user representations for predicting personality labels (e.g., MBTI labels). While recent advances in large language models (LLMs) have improved text encoding capacities, these approaches remain constrained by limited supervision signals due to label scarcity, and under-specified semantic mappings between user language and abstract psychological constructs. We address these challenges by proposing ROME, a novel framework that explicitly injects psychological knowledge into personality detection. Inspired by standardized self-assessment tests, ROME leverages LLMs'role-play capability to simulate user responses to validated psychometric questionnaires. These generated question-level answers transform free-form user posts into interpretable, questionnaire-grounded evidence linking linguistic cues to personality labels, thereby providing rich intermediate supervision to mitigate label scarcity while offering a semantic reasoning chain that guides and simplifies the text-to-personality mapping learning. A question-conditioned Mixture-of-Experts module then jointly routes over post and question representations, learning to answer questionnaire items under explicit supervision. The predicted answers are summarized into an interpretable answer vector and fused with the user representation for final prediction within a multi-task learning framework, where question answering serves as a powerful auxiliary task for personality detection. Extensive experiments on two real-world datasets demonstrate that ROME consistently outperforms state-of-the-art baselines, achieving improvements (15.41% on Kaggle dataset).

Problem

• Detects personality from social media posts using LLMs
• Addresses label scarcity and semantic mapping challenges
• Simulates psychometric questionnaires for interpretable personality evidence

Innovation

• LLMs simulate psychometric questionnaire responses for personality detection
• Question-conditioned Mixture-of-Experts routes post and question representations
• Multi-task learning uses question answering as auxiliary task for prediction