natural language processing

This study systematically investigates whether and how Transformer-based language models acquire syntactic knowledge. Through a large-scale, systematic literature review synthesizing findings from 337 studies and over 3,000 data points, the work presents the first integrated quantitative assessment of syntactic capabilities across multiple languages and model architectures by combining behavioral experiments, representation probing, and mechanistic interpretability methods. The analysis reveals that Transformers possess substantial syntactic knowledge, yet exhibit limitations in phenomena at the syntax–semantics interface and in low-resource languages. It also highlights a pronounced research bias toward English and BERT-family models, with insufficient coverage of linguistic and architectural diversity. This work provides comprehensive empirical evidence and new directions for understanding the mechanisms and boundaries of syntactic generalization in neural language models.

This paper investigates the “linguistic interpretability” of Transformer language models—specifically, whether their internal representations implicitly encode human-like linguistic knowledge. To address this, we systematically review 160 studies, synthesizing cross-lingual and cross-model evidence across four linguistic dimensions: syntax, morphology, lexical semantics, and discourse. Our methodology integrates probing, attribution analysis, and representational similarity comparison, grounded in classical linguistic theory. We thereby bridge critical gaps in multilingual representation analysis and foundational pretraining model interpretation. Results demonstrate that multilingual Transformers consistently encode hierarchical linguistic knowledge, with distinct layers exhibiting functional specialization for specific linguistic phenomena. These findings provide essential theoretical foundations for model diagnostics, controllable text generation, and interdisciplinary research at the intersection of computational linguistics and cognitive neuroscience.

This study addresses critical challenges in multilingual NLP—model bias, insufficient robustness, and difficulty in ethical alignment—by proposing a fine-tuning and deployment framework for large language models (LLMs) targeting low bias and high robustness. Methodologically, it integrates the Hugging Face ecosystem with Transformer architectures, incorporating multilingual tokenization, domain-aware data cleaning and augmentation, and a progressive fine-tuning strategy that jointly optimizes fairness and task performance. Contributions include: (1) a lightweight, cross-lingual fine-tuning paradigm resilient to bias-induced interference; (2) empirical validation across high-stakes domains (e.g., healthcare and finance), demonstrating significant improvements in generalization and fairness for classification and named entity recognition; and (3) an interpretable, auditable, and production-ready LLM deployment pipeline that advances the practical implementation of ethically aligned AI.

The increasing human-likeness of large language model (LLM)-generated text poses growing challenges for reliable authorship attribution and AI content detection. Method: We propose a multi-level textual characterization framework integrating morphological, syntactic, and semantic features with stylistic embeddings. Conducting systematic quantitative analysis across eight domains and eleven mainstream LLMs, we incorporate interpretable linguistic metrics—including dependency distance and sentiment polarity—and combine statistical modeling with controlled sampling strategies. Contribution/Results: We empirically demonstrate that human-written texts exhibit simpler syntactic structures yet higher semantic richness and stylistic diversity, whereas LLM outputs grow increasingly homogeneous across model generations, with diminishing inter-model stylistic divergence. This work provides the first multi-granular, linguistically grounded empirical evidence of fundamental human–machine textual differences, offering interpretable theoretical foundations and actionable technical pathways for text provenance analysis and AI content governance.

To address insufficient integration of lexical distributions and structured knowledge in biomedical named entity recognition (NER), this paper proposes a data-centric knowledge injection paradigm. Specifically, it pioneers the direct conversion of the Unified Medical Language System (UMLS) semantic network and concept hierarchy into structured textual sequences, seamlessly incorporated into BERT’s pretraining pipeline without architectural modifications. The method jointly optimizes masked language modeling and graph-based knowledge reconstruction objectives, enabling co-learning of terminological hierarchical relationships and contextual semantics. A multi-stage pretraining strategy—comprising continued pretraining followed by de novo pretraining—is employed. Empirical evaluation demonstrates significant performance gains across multiple mainstream biomedical NER benchmarks. All models, knowledge serialization code, preprocessing pipelines, and evaluation scripts are publicly released.

Why do large language models (LLMs) require tokenization, and why does character-level modeling lead to performance degradation in Transformers? Method: The authors construct a *k*-order Markov data source and rigorously analyze the cross-entropy of Transformers under character-level versus token-level modeling, grounding the analysis in information-theoretic modeling capacity. They establish a provable relationship between tokenization strategies and the accuracy of sequence probability estimation. Contribution/Results: Theoretically, without tokenization, Transformers collapse to modeling only unigram character distributions, failing to capture higher-order dependencies; with appropriate tokenization, learning single-step token predictions suffices to near-optimally model the source distribution. Empirically, tokenization significantly reduces cross-entropy on high-order Markov sources. This work provides the first rigorous information-theoretic and probabilistic justification that tokenization is a necessary condition for overcoming the fundamental limitations of character-level Transformer modeling.

This study investigates how the depth of Byte Pair Encoding (BPE) tokenization reshapes fundamental statistical properties of natural language—such as Zipf’s law and Heaps’ law—and influences the learning capacity of Transformer models. By theoretically deriving the Zipfian distribution of token frequencies under BPE and the expected slot entropy, and integrating Shannon entropy analysis with Transformer training dynamics, the work establishes the first quantitative link between BPE depth and linguistic statistical regularities. Experimental results demonstrate that recursive application of BPE yields token frequencies that more closely adhere to Zipf’s power law, improves alignment between model-predicted entropy and theoretical expectations, and attenuates local dependencies, driving sequences toward a weakly correlated state. These findings reveal that BPE functions not merely as a compression mechanism but as a statistical transformation that actively restructures the informational architecture of language.

Current evaluations of large language models predominantly emphasize factual accuracy and task-specific performance, often overlooking the linguistic human-likeness of their generated text. This work proposes the first register-aware linguistic evaluation framework, which systematically assesses the naturalness of model outputs by comparing the distributions of 67 Biber lexico-grammatical features between model-generated text and human-written corpora within specific registers, using Maximum Mean Discrepancy as the divergence metric. An evaluation across five English registers on seven open-source instruction-tuned models reveals that all models significantly deviate from human baselines, with the best-performing model varying by register. These findings indicate that register-specific factors account for differences in human-likeness more substantially than model scale alone.

This work addresses the lack of systematic and reproducible frameworks for natural language processing (NLP) research and deployment in low-resource languages by proposing an open-source, end-to-end practical pipeline that spans the full modern NLP workflow. Built around a unified corpus across twelve experimental stages, the framework integrates subword tokenization, vectorization, large model fine-tuning, retrieval-augmented generation, and reinforcement learning from human feedback, all implemented within the Hugging Face ecosystem using openly licensed models to avoid reliance on proprietary APIs. The project delivers the first publicly available tokenizers, embeddings, lexicons, and transliteration benchmarks for languages such as Tajik and Tatar, forming a comprehensive educational curriculum tailored for advanced undergraduates, graduate students, and practitioners, thereby significantly advancing reproducible research and capacity building in low-resource language NLP.

This study addresses machine translation for Bambara—a representative low-resource African language with approximately 14.2 million speakers—where data scarcity severely limits model performance. Method: We systematically compare three Transformer-based paradigms: (i) training from scratch, (ii) fine-tuning a large language model (LLaMA3), and (iii) student–teacher knowledge distillation leveraging LaBSE and BERT-enhanced architectural extensions. We propose a novel cross-lingual knowledge distillation framework tailored to low-resource settings to strengthen semantic representation learning. Experiments are conducted on the Bayelemagaba benchmark and a newly constructed Yiri dataset, evaluated using BLEU and chrF. Contribution/Results: The baseline Transformer achieves the best performance under low-resource constraints, attaining 33.81 BLEU and 41.0 chrF on Yiri. Simpler architectures consistently outperform complex fine-tuning approaches, highlighting a critical trade-off between model simplicity and data efficiency in low-resource MT.

Large language models (LLMs) suffer from low reliability in text classification, while manual annotation remains prohibitively costly. Method: This paper proposes a structured human-AI collaborative classification framework integrating chain-of-thought prompting, few-shot learning, and abductive reasoning to explicitly surface and reconcile divergent human and model judgments via natural-language interaction. It innovatively adapts qualitative research’s collaborative paradigms to quantitative human-AI cooperation, enhancing decision transparency and interpretability. Results: Evaluated on classifying 1,934 pharmaceutical alliance press releases, the framework significantly reduces human annotation effort while achieving higher accuracy and inter-rater consistency than either fully human or LLM-only baselines. It offers a reusable methodological pathway for trustworthy AI-assisted research in the humanities and social sciences.