Large language models for crowd decision making based on prompt design strategies using ChatGPT: models, analysis and challenges

2024-03-22

arXiv.org

AI Summary

This study addresses the challenge of extracting and quantitatively evaluating collective decision-making (CDM) opinions from social media text. Methodologically, it proposes a ChatGPT-based, prompt-driven CDM framework: (i) it pioneers modeling large language models (LLMs) as flexible semantic reasoning modules integrable into CDM workflows; and (ii) it introduces an ontology-guided, structured prompting strategy enabling multi-criteria decision analysis and end-to-end alternative scoring. The contributions include a systematic characterization of LLM limitations in decision consistency, sensitivity, and interpretability. Empirical validation on the real-world TripR-2020Large dataset (TripAdvisor) demonstrates that ChatGPT-generated opinion rankings and composite scores significantly outperform conventional methods—achieving markedly higher accuracy and aligning closely with expert judgment tendencies.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Sentiment analysis and opinion mining

Abstract

Social Media and Internet have the potential to be exploited as a source of opinion to enrich Decision Making solutions. Crowd Decision Making (CDM) is a methodology able to infer opinions and decisions from plain texts, such as reviews published in social media platforms, by means of Sentiment Analysis. Currently, the emergence and potential of Large Language Models (LLMs) lead us to explore new scenarios of automatically understand written texts, also known as natural language processing. This paper analyzes the use of ChatGPT based on prompt design strategies to assist in CDM processes to extract opinions and make decisions. We integrate ChatGPT in CDM processes as a flexible tool that infer the opinions expressed in texts, providing numerical or linguistic evaluations where the decision making models are based on the prompt design strategies. We include a multi-criteria decision making scenario with a category ontology for criteria. We also consider ChatGPT as an end-to-end CDM model able to provide a general opinion and score on the alternatives. We conduct empirical experiments on real data extracted from TripAdvisor, the TripR-2020Large dataset. The analysis of results show a promising branch for developing quality decision making models using ChatGPT. Finally, we discuss the challenges of consistency, sensitivity and explainability associated to the use of LLMs in CDM processes, raising open questions for future studies.

Problem

• Exploring ChatGPT for Crowd Decision Making via sentiment analysis
• Integrating ChatGPT to infer opinions from social media texts
• Assessing ChatGPT's consistency and explainability in decision processes

Innovation

• ChatGPT prompt design for crowd decision making
• Multi-criteria decision making with category ontology
• End-to-end CDM model for opinion scoring