Examining Human-AI Collaboration for Co-Writing Constructive Comments Online

2024-11-05

arXiv.org

AI Summary

This study investigates how large language models (LLMs) can support users in composing constructive online comments on divisive topics—specifically Islamophobia and homophobia—and uncovers systematic human–AI discrepancies in conceptualizing “constructiveness.” Method: A cross-cultural controlled experiment with 600 participants from the U.S. and India integrated behavioral metrics, multidimensional text analysis (sentiment, toxicity, logical coherence), and culturally grounded design. Contribution/Results: We provide the first empirical evidence that humans prioritize logical accuracy and factual grounding, whereas LLMs emphasize politeness and stance balance—revealing a structural judgment misalignment. Human–AI co-writing significantly enhances comment constructiveness, positivity, and safety; better preserves original intent; and reduces toxicity—albeit with occasional minor losses in semantic nuance. We introduce the first LLM-augmented collaborative editing framework explicitly designed for constructive dialogue, offering both theoretical foundations and practical implementation pathways for AI-supported rational public discourse.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper examines how large language models (LLMs) can help people write constructive comments on divisive social issues and whether the notions of constructiveness vary between humans and LLMs. Through controlled experiments with 600 participants from India and the US, who reviewed and wrote constructive comments on online threads on Islamophobia and homophobia, we observed potential misalignment in how LLMs and humans perceive constructiveness in online comments. While the LLM was more likely to prioritize politeness and balance among contrasting viewpoints when evaluating constructive comments, participants emphasized logic and facts more than the LLM did. Despite these differences, participants rated both LLM-generated and human-AI co-written comments as significantly more constructive than those written independently by humans. Our analysis also revealed that LLM-generated comments exhibited more linguistic features associated with constructiveness compared to human-written comments on divisive topics. When participants used LLMs to refine their comments, the resulting comments were more constructive, more positive, and less toxic, retained the original intent but occasionally lost nuances. Based on these findings, we discuss ethical and design considerations in using LLMs to facilitate constructive discourse online.

Problem

• Human-AI alignment in defining constructive comments
• Effectiveness of LLMs in enhancing comment constructiveness
• Ethical implications of LLMs in online discourse

Innovation

• LLMs enhance comment constructiveness via politeness
• Human-AI co-writing reduces toxicity in comments
• LLMs retain intent but may lose nuances