LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models

2026-03-02

AI Summary

Existing reinforcement learning methods struggle to directly optimize diffusion-based large language models due to their reliance on intractable exact likelihood computations, which yield high-variance gradient estimates. This work proposes a likelihood-free policy optimization framework that introduces vector field flow matching into discrete token spaces, enabling direct optimization of denoising logits via contrastive learning while incorporating intermediate-state consistency constraints to enhance generation quality. To circumvent approximation errors in likelihood estimation, the method further designs a geometric velocity correction mechanism that enables accurate gradient estimation. Additionally, it leverages probability flow straightening to substantially reduce the number of inference steps. Experiments demonstrate that the proposed approach outperforms state-of-the-art methods on code and reasoning benchmarks while achieving approximately 20% faster inference.

Technology Category

• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved remarkable success in improving autoregressive models, especially in domains requiring correctness like mathematical reasoning and code generation. However, directly applying such paradigms to Diffusion Large Language Models (dLLMs) is fundamentally hindered by the intractability of exact likelihood computation, which forces existing methods to rely on high-variance approximations. To bridge this gap, we propose Likelihood-Free Policy Optimization (LFPO), a native framework that maps the concept of vector field flow matching to the discrete token space. Specifically, LFPO formulates alignment as geometric velocity rectification, which directly optimizes denoising logits via contrastive updates. This design effectively bypasses the errors inherent in likelihood approximation, yielding the precise gradient estimation. Furthermore, LFPO enforce consistency by predicting final solutions from intermediate steps, effectively straightening the probability flow to enable high-quality generation with significantly fewer iterations. Extensive experiments demonstrate that LFPO not only outperforms state-of-the-art baselines on code and reasoning benchmarks but also accelerates inference by approximately 20% through reduced diffusion steps.

Problem

• Likelihood-Free
• Policy Optimization
• Diffusion Models
• Reinforcement Learning
• dLLMs

Innovation

• Likelihood-Free Policy Optimization
• Diffusion Large Language Models
• Flow Matching
• Contrastive Updates
• Probability Flow Straightening