Decoding Funded Research: Comparative Analysis of Topic Models and Uncovering the Effect of Gender and Geographic Location

2025-10-21

AI Summary

This study investigates the evolution of research themes in 18 years of funding data from Canada’s Natural Sciences and Engineering Research Council (NSERC), and how demographic–geographic factors—including gender and region—systematically shape thematic distribution. To address the lack of interpretability in topic modeling for fairness assessment, we propose COFFEE, a novel covariate effect estimation algorithm enabling BERTopic to perform statistically grounded, interpretable attribution to demographic variables. We integrate LDA, structural topic modeling (STM), and BERTopic—enhanced via domain-adapted pre-trained language models—to improve thematic granularity and semantic coherence. Results show BERTopic outperforms alternatives in detecting emerging topics (e.g., artificial intelligence); provincial research specialization is highly heterogeneous; and systematic gender-based thematic preferences persist across interdisciplinary fields. The work advances methodological rigor for equity-aware science policy, delivering both a scalable analytical framework and empirical evidence to support fair, efficient, and data-driven research funding decisions.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Web Mining and Content Analysis: Topic discovery and tracking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Optimizing national scientific investment requires a clear understanding of evolving research trends and the demographic and geographical forces shaping them, particularly in light of commitments to equity, diversity, and inclusion. This study addresses this need by analyzing 18 years (2005-2022) of research proposals funded by the Natural Sciences and Engineering Research Council of Canada (NSERC). We conducted a comprehensive comparative evaluation of three topic modelling approaches: Latent Dirichlet Allocation (LDA), Structural Topic Modelling (STM), and BERTopic. We also introduced a novel algorithm, named COFFEE, designed to enable robust covariate effect estimation for BERTopic. This advancement addresses a significant gap, as BERTopic lacks a native function for covariate analysis, unlike the probabilistic STM. Our findings highlight that while all models effectively delineate core scientific domains, BERTopic outperformed by consistently identifying more granular, coherent, and emergent themes, such as the rapid expansion of artificial intelligence. Additionally, the covariate analysis, powered by COFFEE, confirmed distinct provincial research specializations and revealed consistent gender-based thematic patterns across various scientific disciplines. These insights offer a robust empirical foundation for funding organizations to formulate more equitable and impactful funding strategies, thereby enhancing the effectiveness of the scientific ecosystem.

Problem

• Analyzing research trends using comparative topic modeling approaches
• Developing COFFEE algorithm for covariate analysis in BERTopic
• Investigating gender and geographic influences on funded research

Innovation

• Comparative evaluation of three topic modeling approaches
• Introduced COFFEE algorithm for BERTopic covariate analysis
• Analyzed gender and geographic effects on research funding