data normalization

This study investigates the impact of historical text normalization on dating and localization classification of Middle High German charters. Using a digital archival dataset, we compare the performance of support vector machines (SVM), gradient-boosted trees, and BERT-based Transformer models before and after applying linguistics-driven historical normalization. Results show that normalization significantly reduces dating accuracy—confirming that original orthographic variation encodes irreplaceable diachronic cues—while yielding only marginal gains in localization precision. SVM and gradient-boosted trees consistently outperform Transformers, challenging the assumed necessity of Transformer architectures for small-scale historical document tasks. We propose a “selective normalization” strategy that preserves task-critical historical linguistic features, advocating for task-aware preprocessing rather than uniform normalization. This work contributes both empirical evidence against indiscriminate normalization and a methodological framework for principled feature retention in historical text processing.

This study addresses a critical methodological gap in automatic text recognition (ATR) of ancient scripts, where outputs often fail to balance paleographic fidelity with modern readability. To bridge this divide, the paper introduces the “Pre-Editing Normalization” (PEN) task, formally defining the problem of transforming ATR transcriptions into standardized texts according to editorial conventions while preserving intermediate paleographic representations. The authors construct a silver-standard training set of 4.66 million samples and a gold-standard evaluation set of 1,800 expert-corrected instances. Leveraging a ByT5-based sequence-to-sequence architecture and incorporating aligned Old French–Latin texts from the CoMMA corpus and passim alignments, the proposed model achieves a character error rate (CER) of 6.7% on the PEN task, substantially outperforming existing approaches and effectively reconciling ancient script transcription with digital scholarly editing.

This study addresses the pervasive hallucination generation and information distortion in scientific text simplification. Methodologically, it proposes a multi-source signal fusion framework for integrated hallucination detection and faithfulness-preserving correction. Specifically, it constructs a meta-learning-based ensemble classifier by jointly leveraging BERT-based classification, semantic similarity metrics, natural language inference (NLI) models, and large language model (LLM) reasoning capabilities to enhance hallucination detection robustness. Furthermore, it introduces an LLM-based post-editing mechanism explicitly constrained by the original text to ensure faithful reconstruction of simplified outputs. Evaluated on the CLEF 2025 SimpleText Task 2 benchmark, the framework achieves significant improvements in both hallucination detection accuracy and output credibility. It establishes a novel paradigm for scientific text simplification that effectively balances creativity with factual fidelity.

Arabic pretraining corpora suffer from severe redundancy (nearly 60% token-level duplication) and heterogeneous quality. To address this, we propose a “data reuse over new crawling” paradigm, systematically integrating seven existing public Arabic web datasets. Our pipeline applies Arabic-specific quality filtering, MinHash-based deduplication at both document and sentence levels, multi-source fusion, and metadata alignment. The resulting corpus—currently the largest publicly available, deeply deduplicated Arabic dataset—comprises 178 billion tokens across 179 million documents. Empirical evaluation demonstrates substantial improvements in downstream model training efficiency and generalization performance. This corpus has become the de facto standard training data for multiple open-source Arabic large language models, establishing a new principle in Arabic NLP: rigorous, quality-driven data curation takes precedence over mere scale expansion.

This work addresses the limitation of traditional retrieval systems, which treat document representation as a static preprocessing step and thus struggle to adapt to downstream tasks. The authors propose AutoIndex, a novel framework that formulates document representation construction as a learnable program synthesis problem. AutoIndex dynamically generates retrieval-oriented representations by searching over executable transformation programs—such as slicing, augmentation, and normalization—and iteratively refines them using validation feedback. By integrating proxy-guided program search with retrieval quality evaluation, the method enables explicit optimization of document representations. Evaluated on the CRUMB benchmark across all eight tasks, AutoIndex consistently outperforms the full-document BM25 baseline, achieving average improvements of 8.4% in Recall@100 and 8.3% in nDCG@10, with peak gains reaching 30.5% and 43.6%, respectively.

Scientific formulas inherently encode both syntactic structure and semantic meaning, yet these two aspects exhibit significant misalignment in their native representation spaces, limiting the performance of cross-modal retrieval. This work is the first to systematically uncover the weakly observable correspondence between formula syntax and semantics and proposes an explicit alignment approach. Specifically, it employs a graph neural network to encode syntactic structures and a text encoder to model semantic content, integrating them into a unified representation space through contrastive learning. The resulting aligned representations effectively bridge the modality gap and substantially enhance cross-modal retrieval performance, thereby demonstrating the critical role of explicit representation learning in mathematical formula understanding.

This study addresses the challenge of legally sharing copyrighted annotated corpora, which hinders natural language processing models from capturing the full diversity of real-world data. To overcome this limitation, the authors propose a corpus distribution mechanism based on non-invertible hashing: corpus creators publicly release hashed versions of both source text and annotations, enabling users to recover the annotations by aligning their own licensed copies of the text through the same hash function. The approach integrates text alignment algorithms with version-tolerant strategies to achieve robust cross-version matching and is implemented in an open-source Python toolkit named novelshare. Experiments on multiple editions of novels demonstrate token-level alignment accuracy ranging from 98.7% to 99.79%, confirming the method’s efficiency and practical utility.

This study addresses the challenge that limited annotated data causes models to over-rely on specific phrasings, thereby constraining their generalization ability and impairing accurate extraction of problem and method sentences from scientific papers. To mitigate this form dependency, the authors propose a data augmentation strategy based on formalized expression desensitization and introduce a context-enhanced Transformer architecture. This architecture incorporates a context-aware mechanism to assess token importance and suppress noise. Experimental results demonstrate that the proposed approach achieves macro-F1 improvements of 3.71% and 2.67% on two scientific paper datasets, significantly outperforming baseline methods. Additionally, the study reveals that in-context learning (ICL) with large language models yields suboptimal performance on this task.

This study addresses the lack of systematic investigation into the ordering of preprocessing steps in sentiment analysis, which has constrained model performance and efficiency. Focusing on Twitter data, it presents the first quantitative evaluation of the relative impact and optimal sequencing of key preprocessing techniques—including tokenization, text cleaning, stemming, stopword removal (with negations preserved), and spelling correction. Through comprehensive combinatorial experiments, the work demonstrates that tokenization contributes most significantly to model performance, while spelling correction has the least effect. The identified optimal pipeline—tokenization followed by text cleaning, stemming, and stopword removal—substantially enhances model effectiveness and reduces trial-and-error costs, establishing a reproducible and efficient preprocessing paradigm for sentiment analysis.

This study addresses the challenges posed by statutory citations in German legal texts, which are highly compact, multi-targeted, employ domain-specific abbreviations, and refer to fine-grained provisions, rendering them difficult to process automatically. To tackle this, the work presents the first open-source toolchain encompassing the full processing pipeline—comprising a citation parser, a normalizer, and a structured corpus of federal statutes—integrating natural language processing with hierarchical legal modeling to achieve end-to-end structured mapping from raw citations to precise legal provisions. Evaluated on 2,944 annotated citations, the system demonstrates strong performance under strict matching and information extraction metrics; normalized citations significantly outperform simple string matching, and the approach achieves high-fidelity deduplication through reliable clustering of real-world citation variants.