Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective

2026-05-15

AI Summary

This work addresses the lack of systematic research on reinforcement learning (RL) fine-tuning for encoder-decoder machine translation models, which has predominantly focused on decoder-only architectures and relied on parallel corpora. We present the first reference-free RL fine-tuning approach applied to NLLB-200 models (600M and 1.3B parameters), leveraging Group Relative Policy Optimization with reward signals derived from LaBSE and COMET-Kiwi—eliminating the need for target-language references. Experiments across 13 diverse languages demonstrate consistent improvements, with Traditional Chinese achieving up to a 5.03-point gain in chrF++, matching the performance of three rounds of supervised fine-tuning. Our analysis further reveals that the observed gains are closely tied to weak baseline models and highly discriminative rewards, highlighting the method’s particular efficacy in low-resource settings.

Abstract

Production machine translation relies overwhelmingly on encoder-decoder Seq2Seq models, yet reinforcement learning approaches to MT fine-tuning have largely targeted decoder-only LLMs at $\geq$7B parameters, with limited systematic study of encoder-decoder architectures. We apply Group Relative Policy Optimization to NLLB-200 (600M and 1.3B) using a hybrid reference-free reward (LaBSE and COMET-Kiwi) that requires no parallel data at fine-tuning time, evaluating across 13 typologically diverse languages. GRPO yields consistent improvements on all 13 languages, up to $+$5.03 chrF++ for Traditional Chinese, and, without any target-language data, competes with 3-epoch supervised fine-tuning on morphologically complex languages . We identify a consistent empirical pattern in which gains are largest where baseline performance is weakest and reward discriminability is highest, making this approach most effective precisely where parallel data is scarcest, and replicate this pattern across English and Spanish source languages.

Problem

• Reinforcement Learning
• Machine Translation
• Seq2Seq
• Reference-Free
• Fine-Tuning

Innovation

• reference-free reinforcement learning
• encoder-decoder Seq2Seq
• Group Relative Policy Optimization
• hybrid reward
• low-resource machine translation