"Shall We Dig Deeper?": Designing and Evaluating Strategies for LLM Agents to Advance Knowledge Co-Construction in Asynchronous Online Discussions

2025-09-27

AI Summary

Asynchronous online discussions frequently stall early, and existing AI interventions typically target only a single phase, lacking systematic mechanisms to foster collaborative knowledge construction; moreover, the impact of intervention styles remains unclear. This study proposes a phased AI intervention framework featuring dual LLM-based intelligent agents—task-oriented and relationship-oriented—designed through participatory workshops and rigorously evaluated via within-subject experiments across multiple asynchronous discussion rounds. We empirically demonstrate distinct effects: task-oriented interventions enhance content depth (e.g., integrative and critical expression), whereas relationship-oriented interventions improve user experience (e.g., perceived support and participation willingness). Critically, the framework sustains progression from superficial exchange toward deep knowledge integration. These findings provide both theoretical grounding and practical design principles for adaptive collaborative learning agents.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Asynchronous online discussions enable diverse participants to co-construct knowledge beyond individual contributions. This process ideally evolves through sequential phases, from superficial information exchange to deeper synthesis. However, many discussions stagnate in the early stages. Existing AI interventions typically target isolated phases, lacking mechanisms to progressively advance knowledge co-construction, and the impacts of different intervention styles in this context remain unclear and warrant investigation. To address these gaps, we conducted a design workshop to explore AI intervention strategies (task-oriented and/or relationship-oriented) throughout the knowledge co-construction process, and implemented them in an LLM-powered agent capable of facilitating progression while consolidating foundations at each phase. A within-subject study (N=60) involving five consecutive asynchronous discussions showed that the agent consistently promoted deeper knowledge progression, with different styles exerting distinct effects on both content and experience. These findings provide actionable guidance for designing adaptive AI agents that sustain more constructive online discussions.

Problem

• Advancing knowledge co-construction in asynchronous discussions
• Addressing stagnation in early discussion phases
• Evaluating AI intervention styles for progression

Innovation

• LLM agent facilitates progressive knowledge co-construction
• Implements task and relationship oriented intervention strategies
• Promotes deeper knowledge progression in asynchronous discussions