Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI

2026-07-10

AI Summary

This study addresses the technical, practical, and ethical challenges of automating keyword annotation for large-scale crowdsourced digital collections. Focusing on the University of Oxford’s World War II crowdsourced archive, “Their Finest Hour Online Archive,” it presents the first systematic comparison of three methodological approaches—named entity recognition, keyword extraction, and topic modeling—under both traditional statistical models and generative AI frameworks. The findings reveal that no single approach is fully adequate for the task; open-weight extractive models offer a favorable balance between performance and accountability, whereas generative AI, despite its potential, introduces significant ethical risks. The project provides empirical evidence and actionable recommendations for implementing responsible, scalable automation in keyword annotation for cultural heritage archives.

Abstract

Identifying and assigning keywords at scale is a technical, practical, and ethical challenge for crowdsourced collections. This article reports the findings of the "Extracting Keywords from Crowdsourced Collections" project, which used the Their Finest Hour Online Archive, a crowdsourced Second World War digital collection hosted by the University of Oxford, as a case study. The project evaluated three Natural Language Processing approaches to automate keyword extraction: Named Entity Recognition, Keyword Extraction, and Topic Modelling. It tested these approaches across a range of artificial intelligence techniques, from traditional statistical methods to modern GenAI neural networks. Our quantitative and qualitative findings indicate that Natural Language Processing approaches offer real potential for keyword extraction at scale in crowdsourced collections, but that no single method offers a complete solution and that model choice significantly shapes results. We argue that in crowdsourced collections, where metadata is the direct product of engagement with living contributors, automated keyword extraction raises distinct stewardship responsibilities that must be addressed alongside technical performance. Open-weight, extractive models emerge from our evaluation as best placed to support responsible deployment, while generative AI, despite its abstractive potential, introduces accountability risks that anyone managing crowdsourced collections should weigh carefully.

Problem

• crowdsourced collections
• keyword extraction
• metadata stewardship
• natural language processing
• responsible AI

Innovation

• keyword extraction
• crowdsourced collections
• natural language processing
• generative AI
• metadata stewardship