Complementary Learning Approach for Text Classification using Large Language Models

2025-12-08

AI Summary

Large language models (LLMs) suffer from low reliability in text classification, while manual annotation remains prohibitively costly. Method: This paper proposes a structured human-AI collaborative classification framework integrating chain-of-thought prompting, few-shot learning, and abductive reasoning to explicitly surface and reconcile divergent human and model judgments via natural-language interaction. It innovatively adapts qualitative research’s collaborative paradigms to quantitative human-AI cooperation, enhancing decision transparency and interpretability. Results: Evaluated on classifying 1,934 pharmaceutical alliance press releases, the framework significantly reduces human annotation effort while achieving higher accuracy and inter-rater consistency than either fully human or LLM-only baselines. It offers a reusable methodological pathway for trustworthy AI-assisted research in the humanities and social sciences.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Application Domains: Humanities & Computational Social Science
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

In this study, we propose a structured methodology that utilizes large language models (LLMs) in a cost-efficient and parsimonious manner, integrating the strengths of scholars and machines while offsetting their respective weaknesses. Our methodology, facilitated through a chain of thought and few-shot learning prompting from computer science, extends best practices for co-author teams in qualitative research to human-machine teams in quantitative research. This allows humans to utilize abductive reasoning and natural language to interrogate not just what the machine has done but also what the human has done. Our method highlights how scholars can manage inherent weaknesses OF LLMs using careful, low-cost techniques. We demonstrate how to use the methodology to interrogate human-machine rating discrepancies for a sample of 1,934 press releases announcing pharmaceutical alliances (1990-2017).

Problem

• Develops a cost-efficient LLM method for text classification
• Integrates human and machine strengths to offset weaknesses
• Analyzes human-machine rating discrepancies in pharmaceutical press releases

Innovation

• Utilizes LLMs cost-efficiently with few-shot learning
• Integrates human abductive reasoning with machine analysis
• Manages LLM weaknesses via low-cost structured methodology