NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts

2025-02-25

AI Summary

This work addresses the longstanding neglect of indigenous Indonesian scripts—particularly low-resource, non-Unicode-encoded scripts—in NLP research. To bridge this gap, we introduce NusaAksara, the first multimodal, multilingual benchmark dedicated to native Indonesian writing systems. It covers eight scripts—including Lampung—and seven languages, supporting five core tasks: image segmentation, OCR, transcription, machine translation, and language identification; all annotations are expert-verified. Our key contribution is the first systematic integration of both visual and textual modalities for indigenous Indonesian scripts, thereby establishing the first evaluation framework for non-Romanized writing systems. Extensive experiments reveal that state-of-the-art models—including GPT-4o, Llama 3.2, PP-OCR, and LangID—achieve near-zero accuracy across multiple tasks, underscoring the benchmark’s rigor and its critical relevance to real-world low-resource script processing.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Indonesia is rich in languages and scripts. However, most NLP progress has been made using romanized text. In this paper, we present NusaAksara, a novel public benchmark for Indonesian languages that includes their original scripts. Our benchmark covers both text and image modalities and encompasses diverse tasks such as image segmentation, OCR, transliteration, translation, and language identification. Our data is constructed by human experts through rigorous steps. NusaAksara covers 8 scripts across 7 languages, including low-resource languages not commonly seen in NLP benchmarks. Although unsupported by Unicode, the Lampung script is included in this dataset. We benchmark our data across several models, from LLMs and VLMs such as GPT-4o, Llama 3.2, and Aya 23 to task-specific systems such as PP-OCR and LangID, and show that most NLP technologies cannot handle Indonesia's local scripts, with many achieving near-zero performance.

Problem

• Preserving Indonesian indigenous scripts in NLP
• Addressing lack of support for local scripts
• Benchmarking multimodal and multilingual tasks

Innovation

• Multimodal benchmark for scripts
• Includes low-resource languages
• Tests diverse NLP models