Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement

2026-09-16

AI Summary

研究通过分析知乎、Quora和Reddit上的117万条回答，发现AI偏好与真实用户参与度间存在差距，并提出OMRA方法减少54.4%的差距。

Abstract

Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with higher engagement match what real users respond to. We study this question using 1.17 million answers to 25,978 questions from Zhihu, Quora, and Reddit, comparing real platform answers and AI-generated answers across four within-question engagement levels. We introduce Ontological Preference Measurement, which represents answers along three dimensions: logic, affect, and expression. We find a systematic gap between AI preference and real user engagement: as target engagement increases, LLMs add more explicit logical structure, while real user engagement is more strongly associated with affective and expressive salience. We call this tendency logic overbinding. Based on this diagnosis, we propose Ontology-Masked Reasoning Autoencoding (OMRA), a controlled intervention that masks and reconstructs over-explained spans while preserving stance, factual content, and coherence. Across four LLM families, OMRA reduces the measured gap by an average of 54.4%. In human evaluation, OMRA wins 62.4% of pairwise preference judgments against matched real platform answers, even though the real answers are more often judged to be human-written.

Problem

• AI Preference
• Real User Engagement
• Large Language Models
• Online Content

Innovation

• Ontological Preference Measurement
• logic overbinding
• Ontology-Masked Reasoning Autoencoding (OMRA)