ParsiPy: NLP Toolkit for Historical Persian Texts in Python

2025-03-22

AI Summary

Historical Persian languages—such as Middle Persian (Parsig)—face persistent NLP challenges, including orthographic complexity, fragmented corpora, and a lack of standardized digital resources. This paper introduces the first Python-based NLP toolkit specifically designed for historical Persian, implementing an end-to-end pipeline that integrates rule-based and statistical methods for tokenization, linguistically informed lemmatization (incorporating phonological constraints and historical linguistic knowledge), part-of-speech tagging, IPA phonemic transcription, and context-aware word embeddings. Key contributions include: (i) the first lightweight, linguistics-driven stemming module tailored to historical Persian; and (ii) the integration of a curated IPA mapping table and domain-specific knowledge base to ensure philological accuracy without compromising computational efficiency. Evaluation on Parsig corpora demonstrates substantial improvements in automatic processing accuracy for ancient texts, enabling robust digital philology and facilitating diachronic linguistic analysis.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

The study of historical languages presents unique challenges due to their complex orthographic systems, fragmentary textual evidence, and the absence of standardized digital representations of text in those languages. Tackling these challenges needs special NLP digital tools to handle phonetic transcriptions and analyze ancient texts. This work introduces ParsiPy, an NLP toolkit designed to facilitate the analysis of historical Persian languages by offering modules for tokenization, lemmatization, part-of-speech tagging, phoneme-to-transliteration conversion, and word embedding. We demonstrate the utility of our toolkit through the processing of Parsig (Middle Persian) texts, highlighting its potential for expanding computational methods in the study of historical languages. Through this work, we contribute to computational philology, offering tools that can be adapted for the broader study of ancient texts and their digital preservation.

Problem

• Develops NLP toolkit for historical Persian texts analysis
• Addresses complex orthography and fragmentary textual evidence
• Enables digital preservation and computational philology studies

Innovation

• NLP toolkit for historical Persian texts
• Modules for tokenization and lemmatization
• Phoneme-to-transliteration conversion feature