Show or Tell? Modeling the evolution of request-making in Human-LLM conversations

2025-08-02

AI Summary

Prior work lacks structured modeling and longitudinal analysis of user requests in human–large language model (LLM) interactions. Method: We propose the novel task of “dialogue query segmentation,” decomposing user inputs into four semantically distinct components—request, role, context, and auxiliary expression—and perform sequence labeling and behavioral pattern tracking on large-scale chat logs. We introduce a diachronic analytical framework to examine temporal evolution in user behavior. Contribution/Results: Our study is the first to empirically demonstrate a dynamic shift from early individual exploration toward collective convergence in user querying behavior; further, LLM capability upgrades significantly reshape prompting patterns, with effects stably observable at the community level. We release the first annotated dataset explicitly designed for behavioral evolution analysis, establishing both a methodological foundation and empirical evidence for understanding human–model co-evolution.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Chat logs provide a rich source of information about LLM users, but patterns of user behavior are often masked by the variability of queries. We present a new task, segmenting chat queries into contents of requests, roles, query-specific context, and additional expressions. We find that, despite the familiarity of chat-based interaction, request-making in LLM queries remains significantly different from comparable human-human interactions. With the data resource, we introduce an important perspective of diachronic analyses with user expressions. We find that query patterns vary between early ones emphasizing requests, and individual users explore patterns but tend to converge with experience. Finally, we show that model capabilities affect user behavior, particularly with the introduction of new models, which are traceable at the community level.

Problem

• Segmenting chat queries into components like requests and roles
• Analyzing differences between human-LLM and human-human request-making
• Tracking how user query patterns evolve with experience and model updates

Innovation

• Segmenting chat queries into multiple components
• Diachronic analysis of user expression patterns
• Tracking model impact on user behavior evolution