Distributional Alignment Games for Answer-Level Fine-Tuning

2026-04-29

AI Summary

This work addresses the computational bottleneck in answer-level fine-tuning arising from the intractable marginalization over an exponentially large space of implicit reasoning paths. To overcome this challenge, the authors propose a Distribution Alignment Game framework that reformulates the original optimization problem as a two-player game between a policy generator and an auxiliary target distribution. By seeking a Nash equilibrium, the method transforms the infeasible marginalization into a tractable projection problem, thereby unifying diverse response generation with self-improvement mechanisms. The approach integrates concepts from game theory, variational inference, and Group Relative Policy Optimization—such as Coherence-GRPO—to achieve both theoretical elegance and practical efficiency. Empirical results on mathematical reasoning benchmarks demonstrate substantial reductions in computational complexity alongside consistent performance gains.

Abstract

We focus on the problem of \emph{Answer-Level Fine-Tuning} (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a \emph{Distributional Alignment Game}. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.

Problem

• Answer-Level Fine-Tuning
• Distributional Alignment
• Language Model Optimization
• Reasoning Paths
• Marginalization

Innovation

• Distributional Alignment Game
• Answer-Level Fine-Tuning
• Nash Equilibrium
• Variational Optimization
• Group Relative Policy Optimization