Discrete Adjoint Matching

2026-02-06

AI Summary

Extending efficient entropy-regularized reward optimization methods from continuous spaces—such as Adjoint Matching—to nondifferentiable discrete generative models remains challenging. This work proposes Discrete Adjoint Matching (DAM), which, for the first time, constructs an adjoint estimator for continuous-time Markov chain–based discrete generative models (e.g., diffusion language models) purely from a statistical perspective, thereby transcending conventional control-theoretic frameworks. DAM establishes a novel and effective adjoint-based optimization pathway for discrete generative models, significantly improving fine-tuning performance on both synthetic tasks and mathematical reasoning benchmarks.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Computation methods for solving entropy-regularized reward optimization -- a class of problems widely used for fine-tuning generative models -- have advanced rapidly. Among those, Adjoint Matching (AM, Domingo-Enrich et al., 2025) has proven highly effective in continuous state spaces with differentiable rewards. Transferring these practical successes to discrete generative modeling, however, remains particularly challenging and largely unexplored, mainly due to the drastic shift in generative model classes to discrete state spaces, which are nowhere differentiable. In this work, we propose Discrete Adjoint Matching (DAM) -- a discrete variant of AM for fine-tuning discrete generative models characterized by Continuous-Time Markov Chains, such as diffusion-based large language models. The core of DAM is the introduction of discrete adjoint-an estimator of the optimal solution to the original problem but formulated on discrete domains-from which standard matching frameworks can be applied. This is derived via a purely statistical standpoint, in contrast to the control-theoretic viewpoint in AM, thereby opening up new algorithmic opportunities for general adjoint-based estimators. We showcase DAM's effectiveness on synthetic and mathematical reasoning tasks.

Problem

• discrete generative modeling
• entropy-regularized reward optimization
• adjoint matching
• continuous-time Markov chains
• non-differentiable state spaces

Innovation

• Discrete Adjoint Matching
• entropy-regularized optimization
• discrete generative models
• continuous-time Markov chains
• adjoint estimation