HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training

2025-07-14

AI Summary

To address semantic ambiguity in Korean large language models (LLMs) arising from homophonic Sino-Korean words indistinguishable in Hangul orthography, this paper proposes HanjaBridge: a continual pretraining method that introduces candidate Hanja characters as explicit semantic anchors, preserving all plausible Hanja forms for homographic words to strengthen contextual disambiguation. To mitigate catastrophic forgetting, token-level knowledge distillation is employed. The approach enables zero-overhead deployment—no inference-time modifications or auxiliary modules are required. On the KoBALT benchmark, HanjaBridge achieves a 21% relative improvement, significantly enhancing Korean language understanding. Moreover, it fosters cross-lingual semantic alignment between Korean and Chinese, demonstrating strong transferability. The core innovation lies in leveraging Hanja as a lightweight, interpretable semantic enhancement signal—effectively balancing model performance, training stability, and deployment efficiency.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large language models (LLMs) often show poor performance in low-resource languages like Korean, partly due to unique linguistic challenges such as homophonous Sino-Korean words that are indistinguishable in Hangul script. To address this semantic ambiguity, we propose HanjaBridge, a novel meaning-injection technique integrated into a continual pre-training (CPT) framework. Instead of deterministically mapping a word to a single Hanja (Chinese character), HanjaBridge presents the model with all possible Hanja candidates for a given homograph, encouraging the model to learn contextual disambiguation. This process is paired with token-level knowledge distillation to prevent catastrophic forgetting. Experimental results show that HanjaBridge significantly improves Korean language understanding, achieving a 21% relative improvement on the KoBALT benchmark. Notably, by reinforcing semantic alignment between Korean and Chinese through shared Hanja, we observe a strong positive cross-lingual transfer. Furthermore, these gains persist even when Hanja augmentation is omitted at inference time, ensuring practical efficiency with no additional run-time cost.

Problem

• Resolving semantic ambiguity in Korean LLMs via Hanja
• Improving low-resource language performance in Korean
• Enhancing cross-lingual transfer through Hanja alignment

Innovation

• HanjaBridge injects meaning via Hanja-augmented pre-training
• Uses token-level distillation to prevent catastrophic forgetting
• Enables cross-lingual transfer through shared Hanja semantics