Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation

2025-04-24

AI Summary

Traditional topic modeling methods suffer from poor interpretability and limited capacity to address domain-specific research questions in the social sciences. Method: This paper proposes a large language model (LLM)-based data augmentation framework that integrates controllable semantic text generation into unsupervised topic modeling. Leveraging GPT-4 to synthesize domain-relevant textual data, the approach couples generated corpora with LDA and BERTopic for guided, question-oriented topic discovery—requiring minimal human intervention. A political science–specific corpus and evaluation framework are constructed to support rigorous validation. Contribution/Results: Experiments demonstrate substantial improvements in topic interpretability and task relevance; the method enables direct answering of domain research questions and reduces manual annotation effort by over 70%. By bridging generative AI with social science–driven topic modeling, this work establishes a novel paradigm for theory-informed, question-centered thematic analysis.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Topic discovery and tracking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Unsupervised machine learning techniques, such as topic modeling and clustering, are often used to identify latent patterns in unstructured text data in fields such as political science and sociology. These methods overcome common concerns about reproducibility and costliness involved in the labor-intensive process of human qualitative analysis. However, two major limitations of topic models are their interpretability and their practicality for answering targeted, domain-specific social science research questions. In this work, we investigate opportunities for using LLM-generated text augmentation to improve the usefulness of topic modeling output. We use a political science case study to evaluate our results in a domain-specific application, and find that topic modeling using GPT-4 augmentations creates highly interpretable categories that can be used to investigate domain-specific research questions with minimal human guidance.

Problem

• Improving interpretability of topic models for social science research
• Enhancing topic modeling for targeted domain-specific questions
• Reducing human guidance needed in topic model analysis

Innovation

• LLM-generated text augmentation improves topic modeling
• GPT-4 creates interpretable categories for research
• Minimal human guidance needed for domain-specific questions