🤖 AI Summary
This work addresses the lack of systematic and reproducible frameworks for natural language processing (NLP) research and deployment in low-resource languages by proposing an open-source, end-to-end practical pipeline that spans the full modern NLP workflow. Built around a unified corpus across twelve experimental stages, the framework integrates subword tokenization, vectorization, large model fine-tuning, retrieval-augmented generation, and reinforcement learning from human feedback, all implemented within the Hugging Face ecosystem using openly licensed models to avoid reliance on proprietary APIs. The project delivers the first publicly available tokenizers, embeddings, lexicons, and transliteration benchmarks for languages such as Tajik and Tatar, forming a comprehensive educational curriculum tailored for advanced undergraduates, graduate students, and practitioners, thereby significantly advancing reproducible research and capacity building in low-resource language NLP.
📝 Abstract
This preprint presents a systematic, research-oriented practicum that guides the reader through the entire modern NLP pipeline: from tokenisation and vectorisation to fine-tuning of large language models, retrieval-augmented generation, and reinforcement learning from human feedback. Twelve hands-on sessions combine concise theory with detailed implementation plans, formalised evaluation metrics, and transparent assessment criteria. The work is not a conventional textbook: it is designed as a reproducible research artefact where every session requires publishing code, models, and reports in public repositories. All experiments are conducted on a single evolving corpus, and the work advocates open-weight models over commercial APIs, with special attention to the Hugging Face ecosystem. The material is enriched by original research on low-resource languages, incorporating linguistic resources for Tajik and Tatar (subword tokenisers, embeddings, lexical databases, and transliteration benchmarks), demonstrating how modern NLP can be adapted to data-scarce environments. Designed for senior undergraduates, graduate students, and practising developers seeking to implement, compare, and deploy methods from classical ML to state-of-the-art LLM-based systems.