nlp preprocessing

Conventional text preprocessing techniques—such as stopword removal, lemmatization, and stemming—rely heavily on language-specific linguistic rules and ignore contextual information, limiting their generalizability across multilingual settings. Method: This paper pioneers a systematic investigation of large language models (LLMs) as context-aware, universal preprocessors. Leveraging prompt engineering, we uniformly perform the three preprocessing tasks across six European languages without language-specific annotations or handcrafted rules. Contribution/Results: Experiments show LLMs achieve 97%, 82%, and 74% accuracy on stopword removal, lemmatization, and stemming, respectively. Downstream text classification models fed with LLM-preprocessed inputs attain up to a 6-percentage-point improvement in F1 score. This work demonstrates the feasibility and effectiveness of LLM-driven, end-to-end, context-sensitive, and multilingual-compatible text preprocessing—establishing a novel paradigm that reduces reliance on manual linguistic rules and enhances preprocessing robustness.

Non-standardized text preprocessing leads to unstable ontology matching (OM) results. Method: We systematically evaluate the impact of a four-stage preprocessing pipeline—tokenization, normalization, stopword removal, and stemming/lemmatization—across 49 OAEI ontology alignment tasks. Our analysis reveals that the first stage (tokenization and normalization) contributes significantly more to matching performance than subsequent stages. Based on this finding, we propose a context-driven preprocessing repair mechanism: (i) dynamically constructing a preservation lexicon to suppress spurious alignments; and (ii) integrating large language models (LLMs) into the pipeline via function calling—bypassing prompt engineering to prevent ground-truth drift. Contribution/Results: Evaluated across eight OAEI tracks, our approach substantially improves matching accuracy and demonstrates robustness. It establishes a reproducible, structured paradigm for synergistic integration of LLMs with classical NLP preprocessing in ontology matching.

Traditional Byte-Pair Encoding (BPE) tokenization introduces token redundancy in low-resource languages, degrading the performance of small-scale models. Method: This paper proposes a BPE configuration method integrating hyperparameter optimization and compressed sensing. It systematically searches key BPE hyperparameters—including vocabulary size and merge iterations—and jointly evaluates configurations using intrinsic metrics (e.g., token count) and extrinsic task performance (generation and classification). Contribution/Results: The study provides the first empirical evidence that BPE configuration significantly impacts multilingual modeling for low-resource languages. Experiments across diverse languages and model scales show that optimal configurations reduce token counts by 12.7% on average and improve downstream task accuracy by 1.8–3.4 percentage points for small models. These gains substantially enhance modeling efficiency and generalization capability in low-resource settings.

This study addresses critical challenges in multilingual NLP—model bias, insufficient robustness, and difficulty in ethical alignment—by proposing a fine-tuning and deployment framework for large language models (LLMs) targeting low bias and high robustness. Methodologically, it integrates the Hugging Face ecosystem with Transformer architectures, incorporating multilingual tokenization, domain-aware data cleaning and augmentation, and a progressive fine-tuning strategy that jointly optimizes fairness and task performance. Contributions include: (1) a lightweight, cross-lingual fine-tuning paradigm resilient to bias-induced interference; (2) empirical validation across high-stakes domains (e.g., healthcare and finance), demonstrating significant improvements in generalization and fairness for classification and named entity recognition; and (3) an interpretable, auditable, and production-ready LLM deployment pipeline that advances the practical implementation of ethically aligned AI.

This work addresses high-accuracy conversion of unstructured/semi-structured documents (e.g., contracts, academic papers, invoices) into structured, machine-readable data. Method: We systematically survey and empirically compare modular pipeline approaches against end-to-end multimodal large models, proposing a unified framework integrating OCR, layout analysis (LayoutParser), graph neural networks, vision-language models (VLMs), and specialized formula/table recognition. We identify and characterize core bottlenecks—layout understanding, dense text recognition, and cross-modal alignment—for the first time. Contribution/Results: We establish a comprehensive analytical framework covering methodology, challenges, and benchmarks, revealing >32% performance gaps of current SOTA on complex layouts (e.g., multi-column, nested tables). We propose a “dual-driven” evolution path emphasizing both data diversity and scale, and open-source a larger annotated dataset to significantly advance knowledge base construction and training-data generation for large models.

This work addresses the lack of systematic and reproducible frameworks for natural language processing (NLP) research and deployment in low-resource languages by proposing an open-source, end-to-end practical pipeline that spans the full modern NLP workflow. Built around a unified corpus across twelve experimental stages, the framework integrates subword tokenization, vectorization, large model fine-tuning, retrieval-augmented generation, and reinforcement learning from human feedback, all implemented within the Hugging Face ecosystem using openly licensed models to avoid reliance on proprietary APIs. The project delivers the first publicly available tokenizers, embeddings, lexicons, and transliteration benchmarks for languages such as Tajik and Tatar, forming a comprehensive educational curriculum tailored for advanced undergraduates, graduate students, and practitioners, thereby significantly advancing reproducible research and capacity building in low-resource language NLP.

This work investigates how to effectively leverage pretrained encoder-decoder models to enhance constituent parsing performance, with a focus on both continuous and discontinuous syntactic structures. The authors formulate constituent parsing as a sequence-to-sequence task and present the first systematic exploration of architectures such as BART, mBART, and T5 for this purpose, evaluating multiple tree linearization strategies. Experimental results demonstrate that the proposed approach outperforms existing seq2seq models on continuous constituent parsing, achieving performance on par with state-of-the-art specialized parsers. Moreover, it exhibits strong generalization capabilities on benchmarks involving discontinuous structures, underscoring the potential of pretrained encoder-decoder frameworks to uniformly handle both structural paradigms within a single model.

This study addresses the lack of systematic investigation into the ordering of preprocessing steps in sentiment analysis, which has constrained model performance and efficiency. Focusing on Twitter data, it presents the first quantitative evaluation of the relative impact and optimal sequencing of key preprocessing techniques—including tokenization, text cleaning, stemming, stopword removal (with negations preserved), and spelling correction. Through comprehensive combinatorial experiments, the work demonstrates that tokenization contributes most significantly to model performance, while spelling correction has the least effect. The identified optimal pipeline—tokenization followed by text cleaning, stemming, and stopword removal—substantially enhances model effectiveness and reduces trial-and-error costs, establishing a reproducible and efficient preprocessing paradigm for sentiment analysis.

This paper addresses online dependency parsing by proposing a word-by-word, real-time parsing algorithm aligned with human linguistic cognition. The method dynamically constructs a dependency tree incrementally, assigning each input token its head immediately upon arrival—without backtracking or global reanalysis. Built upon a dynamic programming framework, the algorithm has a theoretical worst-case time complexity of O(n³); however, empirical evaluation shows this bound is attained only for extremely short sentences (n ≤ 15), while for typical sentence lengths (n > 20), practical runtime scales nearly linearly. Its primary contribution lies in unifying cognitive plausibility—specifically, incremental attachment and zero backtracking—with provably polynomial time complexity within a single algorithmic framework. This design substantially improves both accuracy and robustness over conventional greedy online parsers. Moreover, it furnishes a linguistically interpretable and formally verifiable syntactic foundation for neuro-symbolic language processing models.