AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russian

2025-08-13

AI Summary

This study addresses academic integrity challenges posed by AI-generated Russian scientific abstracts in multilingual scholarly contexts, introducing the first AI-generated detection task specifically for Russian research abstracts. Methodologically, we construct a large-scale, multidisciplinary Russian abstract dataset comprising human-written texts and AI-generated samples from five state-of-the-art LLMs—including GPT-4-Turbo and Llama3.3-70B—emphasizing cross-domain and cross-model generalization; detection is modeled using machine learning and NLP techniques. Contributions include: (1) releasing the first benchmark for AI-generated detection in Russian scientific abstracts; (2) organizing an international shared task that attracted 10 participating teams submitting 159 runs, with top-performing systems achieving AUC scores exceeding 0.97; and (3) establishing a continuously maintained, open-access platform to advance research on AI-text detection for low-resource languages.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The rapid advancement of large language models (LLMs) has revolutionized text generation, making it increasingly difficult to distinguish between human- and AI-generated content. This poses a significant challenge to academic integrity, particularly in scientific publishing and multilingual contexts where detection resources are often limited. To address this critical gap, we introduce the AINL-Eval 2025 Shared Task, specifically focused on the detection of AI-generated scientific abstracts in Russian. We present a novel, large-scale dataset comprising 52,305 samples, including human-written abstracts across 12 diverse scientific domains and AI-generated counterparts from five state-of-the-art LLMs (GPT-4-Turbo, Gemma2-27B, Llama3.3-70B, Deepseek-V3, and GigaChat-Lite). A core objective of the task is to challenge participants to develop robust solutions capable of generalizing to both (i) previously unseen scientific domains and (ii) models not included in the training data. The task was organized in two phases, attracting 10 teams and 159 submissions, with top systems demonstrating strong performance in identifying AI-generated content. We also establish a continuous shared task platform to foster ongoing research and long-term progress in this important area. The dataset and platform are publicly available at https://github.com/iis-research-team/AINL-Eval-2025.

Problem

• Detecting AI-generated Russian scientific abstracts to preserve academic integrity
• Addressing limited detection resources in multilingual scientific publishing
• Developing robust solutions for unseen domains and new AI models

Innovation

• Large-scale dataset with 52,305 Russian scientific abstracts
• Detection across 12 domains and five advanced LLMs
• Continuous platform for ongoing AI-content detection research