AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers

2026-01-15

AI Summary

This work addresses the limited performance of large language models on low-resource languages and fine-grained named entity recognition (FgNER) tasks by proposing the first unified, lightweight solution covering 36 languages—including underrepresented ones such as Bodo and Manipuri—and serving a global population of 6.6 billion. The approach integrates a multilingual mixture-of-experts architecture, an intelligent routing agent, and a web-based interactive platform to enable efficient online annotation, offline deployment on edge devices, and user-friendly access for non-technical users. The project publicly releases 49 language-specific expert models, offering sub-second FgNER inference via GitHub and Hugging Face, thereby substantially lowering the barrier to entry for multilingual fine-grained entity recognition.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce AWED-FiNER, an open-source ecosystem designed to bridge the gap in Fine-grained Named Entity Recognition (FgNER) for 36 global languages spoken by more than 6.6 billion people. While Large Language Models (LLMs) dominate general Natural Language Processing (NLP) tasks, they often struggle with low-resource languages and fine-grained NLP tasks. AWED-FiNER provides a collection of agentic toolkits, web applications, and several state-of-the-art expert models that provides FgNER solutions across 36 languages. The agentic tools enable to route multilingual text to specialized expert models and fetch FgNER annotations within seconds. The web-based platforms provide ready-to-use FgNER annotation service for non-technical users. Moreover, the collection of language specific extremely small sized open-source state-of-the-art expert models facilitate offline deployment in resource contraint scenerios including edge devices. AWED-FiNER covers languages spoken by over 6.6 billion people, including a specific focus on vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo. The resources can be accessed here: Agentic Tool (https://github.com/PrachuryyaKaushik/AWED-FiNER), Web Application (https://hf.co/spaces/prachuryyaIITG/AWED-FiNER), and 49 Expert Detector Models (https://hf.co/collections/prachuryyaIITG/awed-finer).

Problem

• Fine-grained Named Entity Recognition
• Low-resource Languages
• Multilingual NLP
• Vulnerable Languages
• Named Entity Recognition

Innovation

• Fine-grained Named Entity Recognition
• Multilingual NLP
• Expert Detector Models
• Agentic Toolkits
• Low-resource Languages