Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations

2026-06-24

AI Summary

This study investigates individual differences among human annotators in coreference resolution and discourse relation recognition, and their impact on the perception of textual coherence. For the first time in Czech, two multi-annotator parallel corpora—comprising 1,024 and 512 contextual instances, respectively—were constructed, with simultaneous collection of annotation decisions and qualitative justifications. The analysis integrates cross-categorical linguistic phenomena and contrasts human judgments with predictions from automatic models. Results reveal inter-annotator agreement rates of 60–65%, with model prediction discrepancies frequently aligning with cases of human annotation difficulty. Annotator comments further uncover substantial variation in cognitive strategies and confidence levels. This work systematically elucidates the individual variability inherent in text comprehension and its underlying cognitive drivers.

Abstract

As previous research on annotator disagreement in discourse phenomena has shown, understanding text coherence varies considerably from one individual to another. To explore this phenomenon, we created two corpora with multiple annotations of Czech texts, accompanied by annotators' explanations of their choices. The first corpus consists of 1,024 contexts annotated in parallel by three annotators. It captures differences in the identification of coreference across various text types and grammatical-semantic categories, including pronouns, full noun phrases, and anaphoric adverbials. The second corpus comprises 512 contexts, annotated in parallel by five annotators, and focuses on identifying discourse relations in attributive and non-attributive constructions. Both corpora achieve a comparable inter-annotator agreement of approximately 60-65%. For coreference annotation, agreement tends to be lower in cases where automatic coreference resolution models disagree, suggesting that when the models disagree, the examples tend to be more difficult or ambiguous for human annotators to interpret. The annotators' comments, both for coreference and discourse relations, further reveal differences in interpretation, varying levels of confidence in text understanding, and individual reading strategies.

Problem

• coreference
• discourse relations
• annotator disagreement
• human label variation
• text coherence

Innovation

• corpus construction
• annotator disagreement
• coreference resolution
• discourse relations
• human annotation rationale