QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums

2024-05-08

arXiv.org

AI Summary

Online forum texts encapsulate rich, real-world community concerns, yet conventional qualitative and quantitative methods struggle to scale while maintaining interpretability for structured insight extraction. This study introduces the first end-to-end, LLM-driven analytical framework tailored to over one million Reddit comments from ride-sharing communities. The framework integrates structured prompt chaining, multi-stage verification-based reasoning, a human-in-the-loop evaluation闭环, and quantitative metric mapping. It innovatively enables automated, fine-grained attribution and quantification of AI-related issues—such as algorithmic transparency and dispatch fairness—to worker concerns. Across >1M comments, the system systematically identifies and quantifies seven core platform-worker concern dimensions, achieving substantial gains in both analytical efficiency and interpretability. This work establishes a novel paradigm for empirical labor rights research and algorithmic platform governance.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Philosophy and Ethics of AI: AI & Jobs/Labor
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Online discussion forums provide crucial data to understand the concerns of a wide range of real-world communities. However, the typical qualitative and quantitative methodologies used to analyze those data, such as thematic analysis and topic modeling, are infeasible to scale or require significant human effort to translate outputs to human readable forms. This study introduces QuaLLM, a novel LLM-based framework to analyze and extract quantitative insights from text data on online forums. The framework consists of a novel prompting and human evaluation methodology. We applied this framework to analyze over one million comments from two of Reddit's rideshare worker communities, marking the largest study of its type. We uncover significant worker concerns regarding AI and algorithmic platform decisions, responding to regulatory calls about worker insights. In short, our work sets a new precedent for AI-assisted quantitative data analysis to surface concerns from online forums.

Problem

• Extract quantitative insights from forums
• Scale analysis of large text data
• Address worker concerns on AI decisions

Innovation

• LLM-based framework
• Quantitative insights extraction
• Novel prompting methodology