🤖 AI Summary
Existing Cohen’s and Fleiss’ kappa statistics are restricted to single-label classification and lack robustness in realistic settings—such as comorbid psychiatric diagnoses or multi-behavior coding—where multi-label annotations, hierarchical category structures, variable numbers of annotators, and missing labels commonly occur. This paper introduces a generalized κ statistic: the first extension of Fleiss’ kappa to multi-label settings; it incorporates a category-weighting matrix and hierarchical distance metrics to capture semantic similarity among labels, and employs probabilistic expectation-based estimation to ensure robustness against missing data and variable annotator counts. Theoretically, it strictly reduces to classical Fleiss’ kappa under single-label conditions. Implemented in R and Excel, the method is validated through rigorous mathematical derivation and empirical application to psychiatric multi-diagnosis data. Its interpretable range is formally established as [−1, 1], with principled guidelines for interpretation.
📝 Abstract
Cohen’s and Fleiss’ kappa are well-known measures of inter-rater agreement, but they restrict each rater to selecting only one category per subject. This limitation is consequential in contexts where subjects may belong to multiple categories, such as psychiatric diagnoses involving multiple disorders or classifying interview snippets into multiple codes of a codebook. We propose a generalized version of Fleiss’ kappa, which accommodates multiple raters assigning subjects to one or more nominal categories. Our proposed documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} egin{document}$$kappa $$end{document}κ statistic can incorporate category weights based on their importance and account for hierarchical category structures, such as primary disorders with sub-disorders. The new documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} egin{document}$$kappa $$end{document}κ statistic can also manage missing data and variations in the number of raters per subject or category. We review existing methods that allow for multiple category assignments and detail the derivation of our measure, proving its equivalence to Fleiss’ kappa when raters select a single category per subject. The paper discusses the assumptions, premises, and potential paradoxes of the new measure, as well as the range of possible values and guidelines for interpretation. The measure was developed to investigate the reliability of a new mathematics assessment method, of which an example is elaborated. The paper concludes with a worked-out example of psychiatrists diagnosing patients with multiple disorders. All calculations are provided as R script and an Excel sheet to facilitate access to the new documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} egin{document}$$kappa $$end{document}κ statistic. Supplementary Information The online version contains supplementary material available at 10.3758/s13428-025-02746-8.