Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild

2025-05-21

AI Summary

This study investigates how users actively guide large language models (LLMs) through multi-turn interactions to generate desired content in complex writing tasks, rather than passively accepting outputs. Method: Leveraging real-world interaction logs from Bing Copilot and WildChat, the authors apply sequence pattern mining, statistical hypothesis testing, and qualitative coding to systematically identify generalizable, prototypical human-AI collaboration behaviors—termed Prototypical Adaptive Task-handling Heuristics (PATHs)—and their statistically significant mappings to writing intentions. Contribution/Results: A small set of PATHs accounts for the majority of interaction variance; users primarily co-construct text via intent revision, textual exploration, stylistic modulation, and iterative questioning. The work shifts LLM alignment research from static output optimization toward dynamic modeling of interactive processes, providing an empirically grounded behavioral framework and design principles for interactive AI systems.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As large language models (LLMs) are used in complex writing workflows, users engage in multi-turn interactions to steer generations to better fit their needs. Rather than passively accepting output, users actively refine, explore, and co-construct text. We conduct a large-scale analysis of this collaborative behavior for users engaged in writing tasks in the wild with two popular AI assistants, Bing Copilot and WildChat. Our analysis goes beyond simple task classification or satisfaction estimation common in prior work and instead characterizes how users interact with LLMs through the course of a session. We identify prototypical behaviors in how users interact with LLMs in prompts following their original request. We refer to these as Prototypical Human-AI Collaboration Behaviors (PATHs) and find that a small group of PATHs explain a majority of the variation seen in user-LLM interaction. These PATHs span users revising intents, exploring texts, posing questions, adjusting style or injecting new content. Next, we find statistically significant correlations between specific writing intents and PATHs, revealing how users' intents shape their collaboration behaviors. We conclude by discussing the implications of our findings on LLM alignment.

Problem

• Analyzing user-LLM interaction behaviors in writing tasks
• Identifying prototypical collaboration patterns (PATHs) in AI-assisted writing
• Exploring correlations between writing intents and collaboration behaviors

Innovation

• Analyzing multi-turn user-LLM interactions in writing workflows
• Identifying prototypical Human-AI collaboration behaviors (PATHs)
• Correlating writing intents with specific collaboration behaviors