Research Entity Extraction and Topic Detection from UKRI Grant Proposals

2026-06-29

AI Summary

This study addresses the need to support public research investment decisions by automatically extracting research entities from UK funding proposals and identifying emerging topics. To this end, we develop a three-stage pipeline that first employs large language models to extract research entities and then maps them to the OpenAlex Topics ontology for thematic classification. We present the first systematic evaluation of Mistral, GPT-4o, and the DSIT-Taxonomies algorithm on sensitive research-related texts. Results show that Mistral achieves a topic classification accuracy of 90.5%, significantly outperforming DSIT-Taxonomies (71.4%) and matching GPT-4o’s performance, while offering superior deployment security and computational efficiency. These findings highlight the strong potential of open-source models for applications in science policy and research intelligence.

Abstract

This paper presents preliminary findings from a UKRI-funded Metascience project comparing three LLM-based approaches, GPT-4o, Mistral, and a bespoke algorithm, DSIT-Taxonomies, for extracting and classifying research entities from funding proposals. Our project "Tracking Stars and Unicorns" aims to identify early signals of emerging research areas to inform public investment. Our methodology employed a three-stage pipeline, leveraging Mistral for primary entity extraction and mapping against the OpenAlex Topics taxonomy. We evaluated our approach across 42 proposals' abstracts from different areas and observed that Mistral and GPT-4o produce comparable, high-quality entity sets with significant semantic overlap, outperforming the fragmented DSIT-Taxonomies approach. Crucially, the Mistral-based approach achieved superior topic classification accuracy (90.5%) compared to the full DSIT-Taxonomies pipeline (71.4%). We conclude that Mistral offers a high-performance, operationally efficient, and secure solution for large-scale analysis of sensitive grant data.

Problem

• Research Entity Extraction
• Topic Detection
• Grant Proposals
• Emerging Research Areas
• Metascience

Innovation

• LLM-based entity extraction
• research topic detection
• Mistral
• OpenAlex Topics
• metascience