Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

2026-06-24

AI Summary

Humanitarian organizations often struggle to consistently and efficiently analyze qualitative data from affected populations due to limited human and technical resources. This study addresses this challenge by constructing a benchmark dataset comprising 150 high-fidelity synthetic interviews and presents the first fine-grained reliability evaluation of 46 large language models (LLMs) in a humanitarian context. Employing Krippendorff’s α, error-type analysis, and domain-specific criteria—such as discrimination detection and needs hierarchy—the work introduces a hierarchical supervision and theme-specific evaluation framework. Results indicate that certain LLMs achieve expert-level performance in deductive coding tasks, yet exhibit instability when handling indirect expressions, out-of-category needs, and protection-sensitive topics, thereby underscoring the necessity of human oversight and domain adaptation.

Abstract

Data from affected populations are crucial for informing humanitarian response, but their value depends on timely and consistent interpretation of nuanced accounts of need. Humanitarian organizations often lack the staff, time, and specialist expertise required to analyze this information at scale. Large language models (LLMs) may expand this capacity, but their reliability for coding qualitative humanitarian data has not been directly established. This benchmark study compares 46 LLMs to a human Gold Standard using 150 high-fidelity synthetic humanitarian transcripts. Evaluation combined inter-rater reliability testing with Krippendorff's alpha, discrepancy analysis distinguishing correct, near-correct, and incorrect codes, and qualitative assessment across humanitarian-specific criteria including discrimination, complex needs hierarchies, and non-standard communication styles. The authors find that multiple LLMs can perform deductive coding at reliability levels comparable to experienced human coders, especially when structured prompts and reasoning-enabled configurations are used. At the same time, aggregate reliability metrics alone are insufficient for deployment decisions. Models varied in recognizing needs expressed indirectly, needs outside predefined categories, and protection-relevant concerns such as physical safety and discrimination. These findings suggest that LLMs can materially expand humanitarian analytical capacity, but not as substitutes for human judgment. Appropriate use requires structured codebooks, reasoning-enabled models, attention to theme-specific performance, and tiered oversight focused on categories where miscoding would have the greatest programmatic consequences. For sensitive humanitarian data, open-weights models deployed on self-hosted infrastructure may offer a viable path for combining analytical scalability with stronger data governance.

Problem

• large language models
• qualitative data coding
• humanitarian data
• reliability
• data governance

Innovation

• large language models
• qualitative data coding
• humanitarian benchmark
• structured prompting
• Krippendorff's alpha