Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification

2025-07-07

AI Summary

In text classification, identifying and modeling edge cases—samples poorly covered by an initial coding schema—has long suffered from high manual effort and poor generalizability. This paper proposes Co-DETECT, a human-expert–LLM collaborative hybrid active annotation framework: it bootstraps with a minimal coding schema and iteratively discovers and abstracts edge-case handling rules via LLM-assisted auto-labeling, uncertainty-driven sample selection, rule induction, and interactive expert feedback. Its core contribution lies in the first deep integration of LLMs’ semantic understanding with domain experts’ cognitive judgment to enable automated edge-case identification, pattern generalization, and continuous coding schema evolution. User studies and empirical evaluation demonstrate that Co-DETECT significantly improves edge-case detection efficiency (+42%) and classification performance (average F1 gain of 3.8%), while yielding a more compact and generalizable annotation scheme.

Technology Category

• Machine Learning: Active Learning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models (LLMs). Co-DETECT starts with an initial, sketch-level codebook and dataset provided by a domain expert, then leverages the LLM to annotate the data and identify edge cases that are not well described by the initial codebook. Specifically, Co-DETECT flags challenging examples, induces high-level, generalizable descriptions of edge cases, and assists user in incorporating edge case handling rules to improve the codebook. This iterative process enables more effective handling of nuanced phenomena through compact, generalizable annotation rules. Extensive user study, qualitative and quantitative analyses prove the effectiveness of Co-DETECT.

Problem

• Identifies edge cases in text classification using LLMs
• Improves codebook with generalizable edge case descriptions
• Combines human expertise with automatic annotation

Innovation

• Mixed-initiative framework combining human and LLM annotation
• LLM identifies edge cases and improves codebook iteratively
• Compact generalizable rules for nuanced text classification