How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

2026-08-10

AI Summary

This study investigates whether large language models (LLMs) can reliably perform the inherently human task of subjectively evaluating social attractiveness. Grounded in ten psychological and relational constructs, the authors developed a theoretically driven taxonomy of personality profiles spanning three levels of attractiveness. Through three experiments, they systematically compared multiple LLMs against a large human sample in terms of rating consistency, stability, and sensitivity to gender cues. Results indicate that LLMs exhibit high rating stability and consistent hierarchical ordering that generally aligns with human judgments. However, the models display a tendency toward extreme ratings for high- and low-attractiveness profiles and show no sensitivity to gender cues. This work presents the first systematic, theory-based evaluation of LLMs’ capacity for social attractiveness judgment, revealing both their potential and inherent biases.

Abstract

Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.

Problem

• large language models
• social attraction
• subjective evaluation
• persona profiles
• human judgment

Innovation

• social attraction
• large language models
• persona profiles
• human-LLM comparison
• gender bias