Exploring the Impact of Occupational Personas on Domain-Specific QA

2025-05-30

AI Summary

This paper investigates the impact of professional personas on domain-specific question answering (QA) performance, systematically distinguishing between “professional identities” (e.g., scientist) and “professional personalities” (e.g., “scientific mindset”). Leveraging large language models, we employ role-injected prompt engineering and conduct empirical evaluation and ablation studies across multiple scientific QA benchmarks. Results show that professional identity roles yield marginal accuracy gains, whereas professional personality roles consistently degrade performance—revealing that associated cognitive constraints impede efficient retrieval and application of domain knowledge. Our key contribution is challenging the implicit assumption that role relevance implies knowledge efficacy: we demonstrate that effective role design must prioritize domain expertise over generalized personality traits. This finding provides both theoretical grounding and practical guidance for domain-aware, controllable prompt engineering. (149 words)

Technology Category

• Natural Language Processing: Question Answering
• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Recent studies on personas have improved the way Large Language Models (LLMs) interact with users. However, the effect of personas on domain-specific question-answering (QA) tasks remains a subject of debate. This study analyzes whether personas enhance specialized QA performance by introducing two types of persona: Profession-Based Personas (PBPs) (e.g., scientist), which directly relate to domain expertise, and Occupational Personality-Based Personas (OPBPs) (e.g., scientific person), which reflect cognitive tendencies rather than explicit expertise. Through empirical evaluations across multiple scientific domains, we demonstrate that while PBPs can slightly improve accuracy, OPBPs often degrade performance, even when semantically related to the task. Our findings suggest that persona relevance alone does not guarantee effective knowledge utilization and that they may impose cognitive constraints that hinder optimal knowledge application. Future research can explore how nuanced distinctions in persona representations guide LLMs, potentially contributing to reasoning and knowledge retrieval that more closely mirror human social conceptualization.

Problem

• Investigating how occupational personas affect domain-specific QA performance
• Comparing Profession-Based Personas vs Occupational Personality-Based Personas impacts
• Assessing whether persona relevance ensures effective knowledge utilization

Innovation

• Using Profession-Based Personas for domain expertise
• Testing Occupational Personality-Based Personas impact
• Analyzing persona relevance and cognitive constraints