Do Language Models Understand Honorific Systems in Javanese?

2025-02-28

AI Summary

Javanese honorifics (Unggah-Ungguh Basa) are deeply rooted in sociolinguistic hierarchy, yet current language models struggle to accurately recognize and generate context-appropriate honorifics conditioned on speaker, addressee, and referent status. Method: We introduce Unggah-Ungguh, the first fine-grained benchmark for Javanese honorifics, comprising three tasks—honorific classification, Javanese–Indonesian translation, and dialogue generation—and propose a situation-aware consistency evaluation framework that explicitly models sociosemantic dimensions of honorific usage. Contribution/Results: Experiments reveal pervasive neutral/low-register bias across mainstream LLMs, with honorific misuse exceeding 65% in cross-lingual translation and dialogue generation. This work exposes a critical gap in culturally grounded language modeling and provides a reproducible methodology and benchmark resource for evaluating and improving honorific competence in low-resource languages.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The Javanese language features a complex system of honorifics that vary according to the social status of the speaker, listener, and referent. Despite its cultural and linguistic significance, there has been limited progress in developing a comprehensive corpus to capture these variations for natural language processing (NLP) tasks. In this paper, we present Unggah-Ungguh, a carefully curated dataset designed to encapsulate the nuances of Unggah-Ungguh Basa, the Javanese speech etiquette framework that dictates the choice of words and phrases based on social hierarchy and context. Using Unggah-Ungguh, we assess the ability of language models (LMs) to process various levels of Javanese honorifics through classification and machine translation tasks. To further evaluate cross-lingual LMs, we conduct machine translation experiments between Javanese (at specific honorific levels) and Indonesian. Additionally, we explore whether LMs can generate contextually appropriate Javanese honorifics in conversation tasks, where the honorific usage should align with the social role and contextual cues. Our findings indicate that current LMs struggle with most honorific levels, exhibitinga bias toward certain honorific tiers.

Problem

• Assessing language models' understanding of Javanese honorific systems.
• Evaluating cross-lingual LMs in Javanese-Indonesian machine translation.
• Exploring LMs' ability to generate context-appropriate Javanese honorifics.

Innovation

• Developed Unggah-Ungguh dataset for Javanese honorifics
• Assessed LMs on honorific classification and translation
• Explored LM generation of context-appropriate honorifics