Can large language models interpret unstructured chat data on dynamic group decision-making processes? Evidence on joint destination choice

2026-01-09

arXiv.org

AI Summary

Traditional surveys struggle to capture the dynamics of group decision-making, while manual annotation of unstructured chat data to extract explicit and implicit factors is prohibitively costly. This work proposes a structured prompting framework for group decision analysis that leverages large language models (LLMs) to automatically parse group chat logs, progressively extracting choice sets, individual preferences, and driving attributes, thereby transforming unstructured conversations into structured decision data. Through human-annotated validation and error analysis, the study provides the first systematic evaluation of LLMs’ capability boundaries in inferring implicit decision factors embedded in social interactions. Results indicate that LLMs reliably identify explicit factors but still require human oversight when interpreting implicit factors tied to cultural and social norms, thus clarifying both the applicability and limitations of automated analysis in this domain.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Social activities result from complex joint activity-travel decisions between group members. While observing the decision-making process of these activities is difficult via traditional travel surveys, the advent of new types of data, such as unstructured chat data, can help shed some light on these complex processes. However, interpreting these decision-making processes requires inferring both explicit and implicit factors. This typically involves the labor-intensive task of manually annotating dialogues to capture context-dependent meanings shaped by the social and cultural norms. This study evaluates the potential of Large Language Models (LLMs) to automate and complement human annotation in interpreting decision-making processes from group chats, using data on joint eating-out activities in Japan as a case study. We designed a prompting framework inspired by the knowledge acquisition process, which sequentially extracts key decision-making factors, including the group-level restaurant choice set and outcome, individual preferences of each alternative, and the specific attributes driving those preferences. This structured process guides the LLM to interpret group chat data, converting unstructured dialogues into structured tabular data describing decision-making factors. To evaluate LLM-driven outputs, we conduct a quantitative analysis using a human-annotated ground truth dataset and a qualitative error analysis to examine model limitations. Results show that while the LLM reliably captures explicit decision-making factors, it struggles to identify nuanced implicit factors that human annotators readily identified. We pinpoint specific contexts when LLM-based extraction can be trusted versus when human oversight remains essential. These findings highlight both the potential and limitations of LLM-based analysis for incorporating non-traditional data sources on social activities.

Problem

• group decision-making
• unstructured chat data
• joint destination choice
• implicit factors
• social activities

Innovation

• Large Language Models
• group decision-making
• unstructured chat data
• prompting framework
• human annotation