Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages

2025-09-02

AI Summary

This work addresses zero-shot cross-lingual named entity recognition (NER) for low-resource Philippine languages (e.g., Tagalog, Cebuano), avoiding fine-tuning large multilingual language models. Methodologically, it introduces a lightweight meta-pretraining framework that integrates first-order Model-Agnostic Meta-Learning (MAML) into the pretraining objective of a compact decoder-based language model—replacing part of the autoregressive modeling to explicitly enhance generalization across morphosyntactically divergent languages. Additionally, language-specific surface anchors (e.g., Tagalog case markers *si*/*ni*) are leveraged to improve entity boundary detection. Experiments across four model scales demonstrate consistent gains: zero-shot micro-F1 improves by 2–6 percentage points; full fine-tuning further boosts performance by 1–3 points; and training convergence accelerates by up to 8%, substantially alleviating memory and latency constraints.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-constrained settings. We ask whether small decoder LMs can be pretrained so that they adapt quickly and transfer zero-shot to languages unseen during pretraining. To this end we replace part of the autoregressive objective with first-order model-agnostic meta-learning (MAML). Tagalog and Cebuano are typologically similar yet structurally different in their actor/non-actor voice systems, and hence serve as a challenging test-bed. Across four model sizes (11 M - 570 M) MAML lifts zero-shot micro-F1 by 2-6 pp under head-only tuning and 1-3 pp after full tuning, while cutting convergence time by up to 8%. Gains are largest for single-token person entities that co-occur with Tagalog case particles si/ni, highlighting the importance of surface anchors.

Problem

• Zero-shot NER adaptation for low-resource Philippine languages
• Improving small decoder LMs' cross-lingual transfer capabilities
• Addressing memory and latency constraints in multilingual NER

Innovation

• Meta-learning pretraining for zero-shot NER
• Replacing autoregressive objective with MAML
• Improving cross-lingual transfer in low-resource languages