A Large-Scale Dataset and Citation Intent Classification in Turkish with LLMs

2025-09-26

AI Summary

Addressing the challenge of citation intent classification in agglutinative languages such as Turkish, this paper introduces the first large-scale, manually annotated, publicly available Turkish citation intent dataset. We propose a programmable classification framework built on DSPy, which jointly leverages automatic prompt optimization and in-context learning to enhance generalization, while integrating multi-model feature extraction with an XGBoost meta-learner for stacked ensemble classification. Our approach achieves 91.3% accuracy on the held-out test set—significantly outperforming handcrafted prompt baselines. This work constitutes the first high-accuracy, reproducible solution for automated citation intent recognition in Turkish, establishing a new benchmark and methodological paradigm for scholarly text analysis in low-resource languages.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding the qualitative intent of citations is essential for a comprehensive assessment of academic research, a task that poses unique challenges for agglutinative languages like Turkish. This paper introduces a systematic methodology and a foundational dataset to address this problem. We first present a new, publicly available dataset of Turkish citation intents, created with a purpose-built annotation tool. We then evaluate the performance of standard In-Context Learning (ICL) with Large Language Models (LLMs), demonstrating that its effectiveness is limited by inconsistent results caused by manually designed prompts. To address this core limitation, we introduce a programmable classification pipeline built on the DSPy framework, which automates prompt optimization systematically. For final classification, we employ a stacked generalization ensemble to aggregate outputs from multiple optimized models, ensuring stable and reliable predictions. This ensemble, with an XGBoost meta-model, achieves a state-of-the-art accuracy of 91.3%. Ultimately, this study provides the Turkish NLP community and the broader academic circles with a foundational dataset and a robust classification framework paving the way for future qualitative citation studies.

Problem

• Classifying citation intent in Turkish academic texts
• Addressing challenges in agglutinative language citation analysis
• Developing automated prompt optimization for citation classification

Innovation

• Automated prompt optimization using DSPy framework
• Stacked generalization ensemble with XGBoost meta-model
• Programmable classification pipeline for citation intent