Scalable and consistent few-shot classification of survey responses using text embeddings

2025-08-27

AI Summary

In qualitative social science research, manual coding of open-ended survey responses suffers from low efficiency and poor inter-coder reliability, while existing NLP approaches—such as supervised classification, topic modeling, and large language models—rely heavily on extensive labeled data, disrupt interpretive workflows, or yield unreliable outputs. To address these challenges, this paper proposes a few-shot classification framework grounded in pretrained text embeddings. It is the first to integrate few-shot learning with off-the-shelf text embeddings for qualitative social science, preserving compatibility with traditional interpretive practices and supporting both embedding fine-tuning and auditing of existing datasets. Evaluated on 2,899 open-ended responses to a physics concepts survey, the framework achieves Cohen’s Kappa scores of 0.74–0.83 against expert coders—significantly improving coding consistency and scalability. The approach enables efficient, interpretable, and low-label-dependency large-scale qualitative analysis.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Application Domains: Humanities & Computational Social Science
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Qualitative analysis of open-ended survey responses is a commonly-used research method in the social sciences, but traditional coding approaches are often time-consuming and prone to inconsistency. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in qualitative analysis, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework that requires only a handful of examples per category and fits well with standard qualitative workflows. When benchmarked against human analysis of a conceptual physics survey consisting of 2899 open-ended responses, our framework achieves a Cohen's Kappa ranging from 0.74 to 0.83 as compared to expert human coders in an exhaustive coding scheme. We further show how performance of this framework improves with fine-tuning of the text embedding model, and how the method can be used to audit previously-analyzed datasets. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale.

Problem

• Classifying open-ended survey responses efficiently
• Overcoming inconsistency in qualitative data coding
• Scaling analysis without extensive labeled data

Innovation

• Text embedding-based classification framework
• Requires few examples per category
• Fits standard qualitative workflows